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ABSTRACT 


This  thesis  addresses  the  problem  of  optimally  re-assigning  strike  aircraft 
to  targets  in  response  to  the  emergence  of  “pop-ups”  or  time-sensitive  targets. 
The  first  part  of  this  thesis  develops  an  automated  decision  aid  to  rapidly  revise 
the  current  air  tasking  order  (ATO),  so  as  to:  maximize  achievement  of  target 
destruction  goals  (weighted  by  target  priorities),  minimize  attrition  risk  to 
employed  assets,  and  disrupt  the  current  ATO  as  little  as  possible.  The  second 
part  of  the  thesis  develops  a  detailed  test  and  evaluation  plan  to  conduct  a 
comparison  of  two  competing  automated  decision  aids  and  the  current  manual 
reassignment  methods.  Critical  operational  issues,  measures  of  effectiveness 
and  measures  of  performance  were  developed  to  fully  evaluate  operational 
performance.  The  time-sensitive-targeting  decision  aid  was  tested  and  validated 
during  major  air  strike  live  exercises  at  Marine  Aviation  Weapons  and  Tactics 
Squadron  One.  Careful  measurements  comparing  the  re-taskings  recommended 
by  the  decision  aid  against  actual  decisions  demonstrated  that  in  every  case  the 
model's  solutions  were  of  better  or  equal  quality,  maximized  combat  asset 
utilization,  and  were  achieved  significantly  faster. 
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EXECUTIVE  SUMMARY 


The  emergence  of  Time  Sensitive  Targets  (TST)  poses  an  imminent  threat 
to  friendly  forces  and  the  successful  completion  of  the  friendly  commanders 
mission.  Failure  to  decide  and  act  within  a  short  period  of  time,  specified  by  a 
target’s  vulnerability  window,  results  in  loss  of  life  or  the  disruption  of  a  vital  area. 

The  intent  of  this  thesis  is  to  provide  the  tactical  decision  maker  with  a  fast 
and  accurate  automated  decision  aid  that  has  been  successfully  demonstrated  in 
a  live  tactical  environment.  The  decision  aid  develops  an  appropriate  aviation 
response  to  the  emergence  of  a  TST.  There  exists  a  multitude  of  systems  that 
provide  a  commander  with  the  necessary  information  to  develop  feasible  courses 
of  action.  However,  the  determination  of  the  best  asset  to  assign  to  each  target 
is  done  manually,  and  based  solely  on  the  experience  of  the  decision  maker. 

The  decision  aid  employs  optimization  techniques,  and  is  an  adaptation  of 
the  Davi  Castro  Model  [2003]  developed  at  the  Naval  Postgraduate  School.  The 
optimization  model  takes  in  all  available  resource  information  provided  by  the  Air 
Tasking  Order  (ATO),  to  include;  the  number  and  type  of  aircraft,  ordnance  load, 
time  window,  and  the  aircrafts  previously  assigned  task.  Coupled  with 
commander’s  guidance  in  the  form  of  a  prioritized  list  of  targets,  and  assessment 
of  the  threat,  the  decision  aid  determines  which  asset  to  assign  to  each  target 
within  the  vulnerability  window.  Options  considered  by  the  decision  aid  include; 
maintain  the  current  plan,  assign  ground  alert  aircraft,  divert  airborne  missions,  or 
combine  multiple  mission  assets  against  a  single  target.  Assignments  are  made 
by  weighting  the  terms  of  the  objective  function  so  as  to  consider  with  each 
assignment:  target  precedence,  risk  to  aircrew,  the  number  of  required  ATO 
changes,  and  available  asset  range. 

In  order  to  make  a  determination  on  whether  or  not  this  decision  aid  can 
support  the  needs  of  the  commander  a  tactical  evaluation  was  conducted  at 
Marine  Aviation  Weapons  and  Tactics  Squadron  One,  Yuma,  Arizona.  A 
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determination  was  made  based  on  the  results  of  the  Critical  Operational  Issues 
(COI)  listed  below: 

CO1 1.  Target  Selection.  The  decision  aid  properly  conducts  target 

prioritization  of  targets  to  ensure  mission  accomplishment  of  higher 
priority  targets  before  lower  priority  targets. 

COI  2.  Asset  Availability.  The  decision  aid  properly  recognizes  the 
assets  that  are  available  for  a  particular  target. 

COI  3.  Target  Asset  Pairing.  The  decision  aid  properly  recommends 
assets  that  have  the  ability  to  destroy  the  target. 

COI  4.  Mission  Risk  Assessment.  The  decision  aid  accurately  assesses 
the  risk  of  its  proposed  assignments. 

COI  5.  Persistence.  The  decision  aid  minimizes  the  number  of  changes 
to  the  ATO  to  achieve  mission  accomplishment. 

COI  6.  Timeliness.  The  decision  aid  provides  a  proposed  solution  fast 
enough  to  be  effective  when  compared  to  current  methods. 

COI  7.  Options.  The  decision  aid  did  not  provide  the  decision  maker  with 
multiple  alternative  solutions  for  mission  re-tasking  when  available. 

COI  8.  Tactical  Accuracy.  The  decision  aids  output  provide  tactically 
acceptable  assignments. 

COI  9.  Interoperability.  The  decision  aid  operates  correctly  with  the 
current  C4I  architecture. 

CO1 10.  Software  Reliability.  The  decision  aid  operates  continuously 
without  interruption  or  failure  for  a  prolonged  period  of  time. 

Careful  measurements  comparing  the  re-taskings  recommended  by  the 
decision  aid  against  actual  decisions  demonstrated  that  in  every  case  the 
model's  solutions  were  of  better  or  equal  quality,  maximized  combat  asset 
utilization,  and  were  achieved  significantly  faster.  The  results  of  the  evaluation 
lead  to  the  determination  of  that  ten  of  the  eleven  requirements  were  adequately 
met.  Although  positive  comments  were  received  in  regard  to  Human  Factors,  no 
formal  evaluation  of  this  critical  operational  issue  was  performed. 

The  bottom  line;  NPS  has  produced  a  decision  aid  that  can  significantly 
improve  the  combat  effectiveness  against  the  emergence  of  a  TST. 
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I.  INTRODUCTION 


A.  PURPOSE 

This  thesis  has  two  purposes.  The  first  is  to  develop  an  automated 
decision  aid  to  rapidly  revise  the  current  Air  Tasking  Order  (ATO),  so  as  to:  (i) 
maximize  achievement  of  target  destruction  goals  (weighted  by  target  priorities), 
(ii)  minimize  attrition  risk  to  employed  assets,  and  (iii)  disrupt  the  current  ATO  as 
little  as  possible.  The  second  purpose  is  to  develop  a  detailed  test  and 
evaluation  plan  to  conduct  a  multiple  comparison  of  two  automated  decision  aids, 
and  the  current  manual  reassignment  methods  employed  by  the  Marine  Aviation 
Command  and  Control  System  (MACCS).  This  test  plan  was  executed  during 
live  exercises  in  April  2004  at  Marine  Air  Station,  Yuma,  Arizona.  In  addition  to 
the  decision  aid  developed  by  the  Naval  Postgraduate  School  (NPS),  the  Rapid 
Asset  Pairing  Tool  (RAPT)  developed  by  the  Space  and  Naval  Warfare 
Command  (SPAWAR)  participated  in  the  operational  test.  Critical  operational 
issues,  measures  of  effectiveness  and  measures  of  performance  are  developed 
to  fully  evaluate  operational  performance. 

B.  BACKGROUND 

1.  Problem  Statement 

In  today’s  military  aviation  command  and  control  centers,  decision  makers 
are  challenged  with  re-tasking  previously  assigned  attack  assets  in  response  to 
the  emergence  of  higher  priority  targets  or  changes  in  the  tactical  situation.  This 
re-tasking  must  be  conducted  in  a  very  short  period  of  time  so  as  to:  achieve  the 
required  probability  of  kill,  minimize  the  risk  to  aircrew  from  surface-to-air  threats, 
and  limit  the  amount  of  changes  to  the  current  ATO. 

2.  Time  Sensitive  Target  (TST)  Decision  Making  Practices 

TSTs  are  “those  targets  requiring  immediate  response  because  they  pose 
(or  will  soon  pose)  a  danger,  or  are  highly  lucrative,  fleeting  targets  of 
opportunity.”  [JP  3-60]  Time-Sensitive  Targeting  is  distinguished  by  the  fact  that 
it  is  purely  reactive  in  nature.  Predominantly,  these  targets  are  known  to  exist 
but  are  not  yet  located.  Therefore,  TSTs  are  not  included  in  advanced  planning 
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of  missions,  and  need  to  be  dealt  with  as  soon  as  their  locations  are  revealed.  In 
contrast,  the  ATO  is  a  product  of  a  deliberate  planning  process.  The  majority  of 
ATO  assignments  are  preplanned  aviation  missions  against  known  targets  with 
known  locations.  The  current  ATO  is  the  principle  tool  used  in  determining 
available  resources  necessary  to  contend  with  changes  in  the  tactical  scenario, 
such  as  the  emergence  of  TSTs. 

Systems  currently  employed  to  support  assignment  of  aviation  assets  to 
targets  of  this  nature  focus  on  providing  the  decision  maker  situational 
awareness  and  a  means  to  share  information  within  the  tactical  data  network. 
These  systems  are  designed  to  support  the  approval  process  of  a  proposed 
asset  assignment  to  strike  the  TST.  The  determination  or  selection  of  an  asset  is 
largely  done  by  manual  methods  based  on  the  experience  of  the  command  and 
control  officer.  The  following  is  a  general  overview  of  the  issues  a  command  and 
control  officer  must  consider  when  selecting  an  asset  or  assets  for  reassignment 
during  the  targeting  phase  of  the  TST  process. 

1.  Available  Resources.  Decision  makers  require  constant  situational 
awareness  to  determine  potential  reassignment  options 

2.  Weapons  Effectiveness.  The  asset  chosen  by  the  decision  maker  to 
attack  the  target  must  be  able  to  meet  the  specified  destruction  criteria. 

3.  Risk  Assessment.  A  determination  as  to  the  vulnerability  to  each  aircraft 
chosen  to  attack  the  target  must  be  made.  This  is  normally  done  in  the 
form  of  a  risk  assessment  based  on  the  enemy’s  air  defense  capabilities. 
It  is  important  to  point  out  that,  “Specific  TSTs  may  be  such  a  threat  to  the 
force  or  to  mission  accomplishment  that  the  JFC  is  willing  to  accept  a 
higher  level  of  risk  and  attack  the  target  immediately.”  [Commanders 
Handbook  for  Time  Sensitive  Targeting,  2002] 

4.  Associated  Risks  of  Employment.  This  consideration  is  open  to 
interpretation.  This  could  imply  the  minimization  of  risk  to  friendly  ground 
forces  (fratricide),  disruption  of  the  current  ATO,  or  minimization  of 
collateral  damage. 

5.  Method  Selection.  Of  the  options  available,  the  decision  maker  must 
select  the  most  appropriate  platform  considering  all  of  the  criteria 
previously  mentioned. 

6.  Decision.  Given  the  method  selected,  this  task  includes  the  approval  of 
the  method  of  attack,  and  the  transition  of  a  decision  into  action.'' 
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These  techniques  are  developed  through  the  education  and  training  of  our 
decision  makers,  who  are  in  many  cases  brought  in  from  military  specialties  other 
than  command  and  control,  predominantly  aviators.  The  decision  makers’ 
knowledge  of  platform  capabilities  and  the  threat  bring  a  tremendous  wealth  of 
experience  to  the  role.  The  education  and  training  provided  by  institutions  such 
as  MAWTS-1  focus  this  generic  aviation  experience  into  applied  real-time 
decision-making. 

C.  OBJECTIVES 

There  are  two  objectives  of  this  thesis.  The  first  objective  is  the 
development  of  a  Time  Sensitive  Targeting  Decision  Aid  that  supports  the  needs 
of  the  decision  maker  in  a  real-time  tactical  environment. 

The  second  objective  is  the  development  of  a  comprehensive  test  and 
evaluation  plan  to  capture  and  assess  whether  or  not  the  decision  aids  enhance 
the  decision  makers  ability  to  conduct  real-time  re-targeting  of  aviation  assets. 
The  alternative  approaches  to  be  evaluated  include: 

a.  Current  manual  assignment  procedures  based  on  military  judgment 
and  experience  taught  in  the  Weapons  and  Tactics  Instructors  (WTI) 
Course  at  Marine  Aviation  Weapons  and  Tactics  Squadron  1 . 

b.  The  Rapid  Asset  Pairing  Tool  (RAPT),  a  genetic-algorithm  based 
optimization  tool  under  development  by  SPAWAR. 

c.  The  integer-programming  based  optimization  tool  developed  by  Major 
Davi  Castro  and  Prof.  Richard  E.  Rosenthal  of  the  Operations 
Research  Department,  Naval  Postgraduate  School. 

D.  DECISION  AID  SUMMARY 

Since  2001,  the  Space  and  Naval  Warfare  Command  (SPAWAR)  has 
been  developing  a  decision  aid  to  help  reassign  strike  assets  when  high  priority 
targets  emerge  during  the  execution  of  an  Air  Tasking  Order.  These  time 
sensitive  targets  or  “pop-up”  targets,  as  they  are  known,  represent  a  change  in 
the  tactical  situation,  which  requires  dynamic  re-tasking  of  aviation  assets.  NPS 
started  working  on  the  same  problem  in  2002.  Though  SPAWAR  and  NPS  use 
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different  optimization  modeling  approaches  inside  their  decision  aids,  both  seek 
to  assign  aircraft  and  weapon  loadouts  to  targets  with  the  following  objectives: 
achieve  the  required  probability  of  kill,  minimize  the  risk  to  aircrew  from  surface- 
to-air  threats,  and  limit  the  amount  of  changes  to  the  current  Air  Tasking  Order 
(ATO). 

The  underlying  approach  in  SPAWAR’s  decision  aid  is  a  genetic 
algorithm  currently  used  in  the  Rapid  Asset  Pairing  Tool  (RAPT).  [Louis, 2003] 
On  3  October  2003,  RAPT  was  demonstrated  to  Captain  Heckert,  Program 
Manager  for  Naval  Mission  Planning  and  Tomahawk  Command  and  Control 
(PMA-281).  The  demonstration  of  RAPT’s  ability  to  quickly  propose  options  in 
reassigning  aircraft  to  higher  priority  targets  was  received  with  encouragement. 
SPAWAR  was  given  the  go-ahead  to  pursue  current  development  with  an 
emphasis  on  determining  whether  the  results  of  the  current  system  could  be 
considered  tactically  accurate. 

Assisting  SPAWAR  in  development  of  a  real-time  decision  aid  for  aviation 
assets  is  the  Operations  Research  Department  at  the  Naval  Postgraduate 
School.  Under  the  direction  of  Professor  Richard  Rosenthal,  Major  Davi  Castro 
of  the  Brazilian  Air  Force  developed  an  integer-programming  based  means  to 
obtain  a  solution  to  the  stated  problem.  In  December  of  2002,  their  work  resulted 
in  the  Static  and  Dynamic  Optimization  Models  used  in  assigning  assets  to  a 
prioritized  list  of  targets.  For  all  intents  and  purposes,  what  was  developed  can 
be  considered  an  ATO  builder  in  its  simplest  form.  Major  Castro,  even  then, 
realized  the  potential  for  real-time  asset  allocation  through  re-optimization  of  an 
existing  ATO.  With  further  enhancements  to  his  original  Static  Model  for  asset 
allocation,  the  ability  to  apply  this  model  to  real-time  re-targeting  is  ready  to  be 
tested. 

E.  SCOPE  AND  LIMITATIONS 

The  Weapons  Tactics  Instructors  course  provided  a  means  to  achieve  the 

objectives  of  this  thesis,  however  our  evaluation  was  not  the  focus  of  MAWTS-1 . 

Emerging  high  priority  targets  are  part  of  the  exercise  scenario  but  may  be 

limited  in  number.  Additionally,  the  scenario  is  limited  to  operations  in  a  desert 
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environment.  Time  Sensitive  Targets  that  can  emerge  in  populated  areas  are  not 
be  evaluated.  The  plan  allows  for  the  evaluation  of  human  factors  aspects  of  the 
decision  aids.  However,  the  decision  aids  were  operated  primarily  by  the 
developers,  and  therefore,  it  was  not  possible  to  conduct  an  effective  evaluation 
of  the  human  factors  issues. 

F.  THESIS  ORGANIZATION 

Chapter  II  provides  a  more  detailed  insight  into  Time  Sensitive  Targeting. 
Chapter  III  provides  a  description  of  the  model  developed  by  Major  Castro,  and 
the  revisions  made  to  adapt  the  model  to  a  real-time  tactical  environment. 
Chapter  IV  discusses  the  development  of  the  critical  operational  issues  central  to 
the  evaluation.  Chapter  V  provides  a  detailed  analysis  of  data  collected  during 
the  evaluation.  Chapter  VI  provides  conclusions  and  recommendations  based  on 
this  analysis. 
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II.  THE  TIME  SENSITIVE  TARGETING  PROCESS 


A.  INTRODUCTION 


The  Time  Sensitive  Targeting  Process  consists  of  six  phases;  find,  fix, 
track,  target,  engage,  and  assess.  This  chapter  discusses  each  phase,  with 
emphasis  on  Targeting,  the  phase  in  which  the  decision  aids  are  employed. 
Figure  1^,  TST  Process  in  Cyclical  Form,  provides  a  simple  graphic  outlining  the 
phases  of  Time  Sensitive  Targeting  and  the  tasks  associated  with  each.  A 
discussion  of  each  of  these  tasks  is  the  focus  of  this  chapter  and  will  provide  the 
reader  with  the  necessary  background  to  understand  the  functioning  of  the 
decision  aids  presented  in  Chapter  III. 


•Assess 
•Report  results 
•Reattack 
recommendation 


•Order  engagement 
&  transmit  order 
•Monitor/manage 
engagement 
•Strike  (attack  TST) 


•Focus  sensors 
•Loca^ 

•Identify 

•Determine  time  available 


•Prioritize  ISR 
•Maintain  track 
•Update  time  available 


Kev 

•Detect  JP  3-60  &  MTTP  Steo 

■Deconflict  Additional  MTTP  Step 

MTTP  Phase 

•Determine  available  resources 

•Develop  options  to  achieve  desired  effect 

•Weaponeer 

•Satisfy  restrictions 

•Deconflict 

•Target  Area  Clearance 
•Risk  assessment 
•Select  method 
•Decide  (final  approval) 


Figure  1  The  TST  Process  in  Cyclical  Form.  Beside  each  phase  of 
the  process  the  tasks  that  must  be  performed  are  listed. 


2  Time  Critical  Targeting  Brief,  Headquarters  United  States  Air  Force  2002 
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1.  The  Find  Phase 

TSTs  are  usually  targets  known  to  exist  but  have  not  yet  been  located. 
Therefore,  the  first  task  is  to  find  them.  The  “find”  phase,  as  referred  to 
previously  in  Figure  3,  receives  the  focus  of  all  intelligence  collection  assets 
available  on  the  battlefield.  Assets  used  in  finding  TST’s  include,  but  are  not 
limited  to:  Special  Operations  Forces  (SOF),  Airborne  Collection  assets 
(JSTARS,  AWACS,  and  Strike  Aircraft),  and  Space  Based  Collection  Assets. 
Those  individuals  who  are  assigned  the  primary  or  alternate  role  of  detecting 
TST’s  receive  their  direction  from  their  commanders  in  various  forms.  The 
guidance  and  direction  for  TST’s  may  come  in  a  format  as  simple  as  a  prioritized 
list  as  seen  in  the  following  example. 

“The  adversary  is  known  to  possess,  and  has  the  capability  to  employ 
nuclear  WMD  against  the  Joint  Force.  Accordingly  the  following  target  types  are 
designated  Joint  TSTs: 

•  Adversary  activities  deploying  WMD  from  known  storage  areas. 

•  Known  or  suspected  TBM  with  WMD  payload.”  [Commanders 
Handbook  for  Time  Sensitive  Targeting,  2002] 

On  the  other  hand,  the  tools  provided  to  assist  in  the  detection  of  TSTs 
can  be  very  specific,  such  as  the  following  notional  decision  matrix  depicted  in 
Figure  2.  The  manual  decision  aid  is  used  by  operators  throughout  the 
battlefield,  whose  responsibility  it  is  to  assist  in  the  prosecution  of  Time  Sensitive 
Targets.  This  tool  helps  to  clearly  identify  targets  of  this  nature  and  provide 
guidance  as  to  the  appropriate  responses  to  them.  The  first  column  lists  the 
TST’s  in  order  of  precedence.  The  second  column  describes  the  specific  type  of 
target  to  be  found.  The  third  column  dictates  who  has  the  authority  to  approve 
the  striking  of  that  target.  The  fourth  column  denotes  any  special  requirements  in 
regards  to  striking  the  target.  The  fifth  column  denotes  the  acceptable  level  of 
risk  for  fratricide  and  collateral  damage.  The  final  column  provides  amplifying 
comments  in  regards  to  actions  taken. 
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Priority 

TST  Target  Desired  Approvai  Additonai  Accpetabie 

Type  Effect  Authority  Restrictions  Risk  Levei  equiremen  s 

or  Notes 
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JFC-2 

JFC-3 

JFC-4 
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5 
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6 

Weapon  System 
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Leader 

Hi2 

Strike 

Immediately  with 
any  asset. 
Package 
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but  will  go 
without 

Personel  or 
groups  meeting 
X  criteria 

Isolate, 
Capture,  or 
kill 

JFc  or 
Above 

Higher  Level 
Notification 
required  prior 
to  striking 

Hi 

NotifyJFC 
immediately  and 
maintain  sensor 
track.  Package 
recommended, 
threat 
dependent. 

Critical  Weapon 
System  B 

Prevent 

movement 

or  use 

JFC 

MED 

Hazard  Analysis 
required. 
Package 
Required 

Critical  Weapon 
System  C 

Neurtalize 

for 

campaign 

duration 

TST  Col 
Chief 

— 

LOW 

SEAD  Required 

Specific  Key 
ground 

force/equipment 

movement 

Destroy 

JFACC 

LOW 

Convoy  or 
military  vehicles 
approaching 
phase  line  green 

Important 
weapon  system 
D 

Neutralize 

for 

campaign 

duration 

TST  Col 
Chief 

LOW 

SEAD  Required 

Figure  2  Notional  TST  Decision  Matrix.^  Depicts  an  example  of  a  tool 
used  by  operators  to  help  identify  and  determine  the  appropriate  course  of 
action  for  each  TST. 

Once  the  process  has  been  initiated  with  an  emerging  target  the  focus  of 
this  phase  then  becomes  the  determination  of  the  target  as  one  of  four  possible 
classifications.  These  classifications  are;  Probable  TST,  Non-TST,  Unknown,  or 
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not  a  target .4  Each  determination  leads  to  one  of  four  possible  actions  by  the 
decision  maker.  Figure  3,  Find  Phase:  Sequence  of  Conditions  and  Actions^, 
provides  the  sequence  just  described  and  illustrates  the  four  possible  actions. 


Figure  3  Find  Phase:  Sequence  of  Conditions  and  Actions.  Defines  the 
sequence  and  events  that  occur  beginning  with  the  emergence  of  a  target 
and  conciuding  appropriate  action  given  the  determination  of  the  target. 

The  final  output,  or  conclusion  of  this  phase,  is  the  nomination  of  a 
probable  TST  for  further  consideration.  This  in  turn  carries  the  target  into  the 
next  phase  of  the  TST  process. 

2.  The  Fix  Phase 

Once  an  emerging  target  is  determined  to  be  a  probable  TST  ,  the  process 
continues  with  the  “fix”  phase.  This  phase  focuses  on  determining  the  precise 
location  of  the  target.  In  this  phase,  data  may  be  correlated  from  a  variety  of 
sources  to  confirm  not  only  the  location  of  the  target,  but  that  it  is  in  fact  a  TST. 
Aside  from  a  location  determination,  an  additional  product  of  this  phase  is  the 
“vulnerability”  window.®  The  task  associated  with  the  vulnerability  window  as 
depicted  previously  in  Figure  3,  is  to  determine  the  time  available  to  conduct  the 

^  Multi-Service  Tactics,  Techniques  ,  and  Procedures  (MTTP)  for  Time  Sensitive  Targets,  FM 
3-60.1,2004 

®  Multi-Service  Tactics,  Techniques  ,  and  Procedures  (MTTP)  for  Time  Sensitive  Targets,  FM 
3-60.1,2004 


®  Multi-Service  Tactics,  Techniques  ,  and  Procedures  (MTTP)  for  Time  Sensitive  Targets,  FM 
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strike.  For  fixed  targets  this  window  will  obviously  be  larger  than  those  that  are 
mobile.  There  is  no  standard  when  it  comes  to  the  size  of  a  vulnerability  window. 
However,  a  brief  presented  in  2002  by  Brigadier  General  Jim  Morehouse,  USAF, 
provided  an  excellent  example  of  a  vulnerability  window  for  mobile  TSTs.  At  that 
time,  mobile  Time  Sensitive  Targets  were  referred  to  as  Time  Critical  Targets 
(TCT)  by  the  United  States  Air  Force.  Figure  4,  Phase  Allotment  Time,^  below 
illustrates  the  vulnerability  window  with  respect  to  mobile  TSTs.  In  order  for  a 
decision  maker  to  be  effective  against  mobile  time  sensitive  targets  a  proposed 
asset  assignment  must  be  made  within  2  minutes. 


Solid”  Intel 
Prep  of 
Battlefield 


I  Parallel 


i 


Select 

Commit 


I 


Processes 


T=0 


I  Formulate  attack  | 

I  Position  weapon  sys  | 
I  Employ  weapon  | 

I  Weapon  fly-out  I 

5 


30 


Performance  Thresholds 
on  TCT  30  Min  Window 


Itlentification 
Tasking  Timelines 
Decision  Aid  Support 
Target  Engagement  svithin 
Contiimons  IPB 
Streamlined  Tasking  Timeline 


2  mins  from  detect 
,t0  mins  detect  to  \\  pn  impac 
2  mins 

30  mins  of  TCT  declaration 

2  min  to  access  data 

3  mins 


Figure  4  Phase  Allotment  Time.  Defines  the  time  aiiotted  to  each 
phase  within  the  TST  Process. 


^  Time  Critical  Targeting  Brief,  Headquarters  United  States  Air  Force  2002 
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3.  The  Track  Phase 

Prior  to  entering  the  “track”  phase,  the  target  has  been  confirmed  as  a 
TST  and  the  location  verified.  This  phase  will  continue  until  engagement  has 
been  completed.®  It  is  important  to  the  decision  maker  that  situational  awareness 
is  maintained,  and  for  this  reason  the  coordination  and  focusing  of  additional 
battlefield  sensors  may  be  required. 

4.  The  Target  Phase 

Although  each  phase  within  the  process  is  of  equal  importance,  the 
“Target”  Phase  is  more  adaptable  to  time  saving  decision  aids,  as  will  be 
discussed  later  in  Chapter  III.  This  phase,  as  depicted  in  figure  1,  involves  nine 
tasks  that  the  decision  maker  must  perform  in  selecting  the  most  capable  assets 
to  address  the  target.  These  tasks  include: 

1 .  Determine  available  assets. 

2.  Develop  options  to  achieve  desired  effects. 

3.  Weaponeer 

4.  Satisfy  Restrictions 

5.  Deconfliction 

6.  Target  Area  Clearance 

7.  Risk  Assessment 

8.  Select  Method 

9.  Decide  (final  approval) 

a.  Determine  Available  Assets 

Available  resources  are  those  assets,  aviation  or  surface-to-surface 
fires,  which  can  strike  the  TST  within  the  designated  vulnerability  window. 
Aviation  assets  that  may  be  considered  available  consist  of  the  following:  1) 
aircraft  not  currently  assigned  to  targets  and  assigned  to  an  alert  status,  2) 
aircraft  currently  assigned  a  target  on  the  ATO  that  is  of  a  lower  priority,  3) 

®  Multi-Service  Tactics,  Techniques  ,  and  Procedures  (MTTP)  for  Time  Sensitive  Targets,  FM 
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aircraft  complete  within  current  mission  and  have  enough  ordnance  and  fuel  to 
accept  a  new  assignment. 

b.  Options  That  Can  Achieve  Desired  Effects 

Of  the  assets  available  to  conduct  the  attack  on  the  TST,  the 
decision  maker  must  select  the  asset  with  the  proper  ordnance  capable  of 
achieving  the  desired  effects.  The  list  of  available  resources  is  therefore  refined 
to  increase  the  probability  of  success  of  the  higher  priority  mission.  This 
refinement  process  denotes  the  task,  and  develops  options  to  achieve  desired 
effects. 

c.  Weaponeering 

For  each  TST,  the  commander  provides  a  definition  for  the  desired 
effects.  For  example,  if  the  TST  is  a  Multiple  Rocket  Launcher  (MRL)  Battery, 
and  the  desired  effect  is  to  destroy  it,  the  commander  defines  what  destroyed 
means.  This  could  be  stated  as  the  number  of  launchers  required  to  be  rendered 
ineffective.  To  achieve  these  desired  effects  the  decision  maker  conducts  the 
task  of  weaponeering.  Weaponeering  is  the  process  of  determining  the  quantity 
of  weapons,  not  just  the  weapon  type,  required  to  achieve  desired  effects. ^ 

d.  Satisfy  Restrictions 

In  the  next  task,  decision  maker  must  verify  any  restrictions  for 
attacking  the  target  prior  to  engagement.  Engagement  restrictions  vary  with 
each  target  and  are  normally  dependent  on  the  precedence  or  value  of  the  TST. 
These  restrictions  may  include:  1)  Collateral  Damage  Guidance,  2)  Rules  of 
Engagement,  3)  Restricted  Fire  Areas,  and  4)  Fire  Support  Coordination 
Measures. '1°  Restrictions  established  by  the  commander  cannot  be  violated 
when  striking  the  target. 


9  Commanders  Handbook  for  Joint  Time  Sensitive  Targeting,  22  March  2002 
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e. 


Deconfliction 


Deconfliction,  denotes  the  coordination  necessary  to  achieve  safe 
and  effective  execution  of  the  mission.  The  deconfliction  of  multiple  aviation 
assets  within  a  confined  airspace  requires  detailed  integration  and  cross 
component  coordination. 

f.  Target  Area  Clearance 

The  next  task  within  this  phase,  Target  Area  Clearance,  is  simply 
a  determination  of  whether  or  not  the  proposed  asset  for  attack  is  permitted  to 
strike  in  the  location  of  the  TST.  This  determination  is  made  by  the  decision 
maker  during  the  evaluation  of  the  nominated  target  and  is  based  on  the  decision 
makers  approval  authority  given  the  geographic  location  of  the  target. 

g.  Risk  Assessment 

Risk  Assessment  encompasses  all  associated  risks  of  employment 
of  a  targeting  solution.  These  risks  include,  but  are  not  limited  to:  1)  risk  of 
fratricide,  2)  risk  to  attacking  forces  based  on  the  threat,  3)  risk  to  non- 
combatants,  and  4)  risk  of  disruption  to  the  current  plan.'i'i  All  of  these  risks  are 
weighed  against  the  value  assessed  to  the  TST.  These  values  vary,  however;  it 
is  crucial  to  point  out  that  “specific  joint  TSTs  may  be  such  a  threat  to  the  force  or 
mission  accomplishment  that  the  Joint  Force  Commander  is  willing  to  accept  a 
higher  level  of  risk  and  attack  the  target  immediately.”  [Commanders  Handbook 
for  Joint  Time  Sensitive  Targeting,  22  March  2002] 

h.  Select  Method 

The  “target”  phase  continues  with  the  selection  of  the  strike  asset 
by  the  responsible  commander.  It  is  the  culmination  of  all  previous  tasks,  and 
based  on  the  vulnerability  of  the  target  will  most  likely  have  to  be  done  in  short 
period  of  time. 

I.  Decide 

The  final  task  within  the  “target”  phase  is  the  decision  made  by  the 
commander.  This  culminating  point  in  the  process  should  take  approximately 
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three  to  five  minutes  in  duration  and  represents  an  approved  action  to  be  taken  in 
response  to  the  TST. 

5.  The  Engage  and  Assess  Phase 

The  TST  process  concludes  with  the  “engage”  and  “assess”  phase.  The 
engage  phase  is  the  commander’s  implementation  of  the  decision.  This  phase 
involves  contacting  the  assets  selected  and  redirecting  them  from  the  original 
assignment.  Once  the  assets  are  redirected  and  the  attack  has  been  conducted 
the  success  of  the  mission  is  assessed.  If  the  desired  effects  are  not  achieved 
the  process  can  begin  again. 

B.  CONCLUSION 

The  targeting  phase  of  the  TST  process  is  of  special  importance  to  thesis. 
It  is  within  this  phase  that  the  decision  aid  developed  by  NPS  is  applied.  The 
discussions  within  this  chapter  provide  the  necessary  background  to  understand 
the  functioning  of  the  decision  aid  presented  in  Chapter  III. 
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III.  OPTIMIZATION  MODEL  FOR  RETASKING  OF  AIR 

STRIKE  ASSETS 


A.  INTRODUCTION 

The  model  discussed  in  this  chapter  is  a  variation  of  the  static  optimization 
model  previously  developed  by  Major  Davi  Castro. The  new  model  optimizes 
the  assignment  of  aviation  assets  to  a  prioritized  set  of  targets  over  a  short  time 
horizon,  taking  into  account  the  presence  of  surface-to-air  threats.  The 
performance  of  this  model  has  been  tested  during  live  tactical  exercises. 

B.  PROBLEM  DESCRIPTION 

The  problem  addressed  is  the  real-time  re-tasking  of  aviation  assets  in 
response  to  a  sudden  change  in  the  current  tactical  situation.  The  type  of 
change  for  which  the  model  is  primarily  intended  is  the  emergence  of  a  Time- 
Sensitive  Target  (TST)  or  “pop-up.”  A  TST  is  previously  designated  by  the 
commander  as  a  high-priority  target,  but  its  location  is  not  known  when  the  Air 
Tasking  Order  (ATO)  is  created.  Therefore,  TST’s  are  not  included  in  pre¬ 
planned  missions  and  need  to  be  addressed  after  their  locations  are  revealed. 

The  urgency  of  engaging  a  TST  requires  immediate  tasking  within  its 
vulnerability  window,  as  described  in  Chapter  2.  The  only  assets  available  to 
engage  the  TST  are  those  currently  scheduled  to  perform  other  missions  or  to  be 
in  an  alert  status  within  the  vulnerability  window.  The  problem  is  then  to  assign 
these  available  assets  to  the  TST  and  to  the  targets  previously  scheduled  for 
attack  in  the  vulnerability  window.  There  is  not  enough  time  to  get  additional 
assets  ready  nor  to  reconfigure  the  ordnance  of  the  available  aircraft.  Assigned 
aircraft  must  be  available  in  the  vulnerability  window  and  capable  of  destroying 
their  targets  to  the  degree  required  within  acceptable  limits  of  attrition  risk. 

Time-critical  conditions  lead  to  a  restricted  set  of  options  for  assigning 
assets  to  targets.  Each  target  included  in  this  problem  must  be  dealt  with  in  one 
of  the  following  ways: 


Optimization  Models  for  Allocation  of  Air  Strike  Assets  with  Persistence,  Davi  Castro,  2003 
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1 .  Maintain  the  current  plan.  That  is,  for  a  non-time-sensitive  target, 
use  the  assets  already  scheduled  to  attack  it  on  the  ATO.  This 
would  be  the  most  convenient  solution  from  a  command  and  control 
perspective,  but  it  may  not  be  feasible  for  all  targets  due  to  the 
added  demand  of  the  TST. 

2.  Employ  assets  on  alert  status.  This  option  would  be  the  most 
convenient  way  to  address  the  TST,  provided  the  assets  on  alert 
are  capable  of  attacking  the  TST  with  sufficiently  high  probability  of 
success  and  low  probability  of  attrition. 

3.  Redirect  the  assets  of  a  previously  scheduled  mission  to  a  different 
target.  When  missions  are  redirected,  there  is  not  enough  time  to 
reorganize  the  aircraft  assigned  to  the  previously  scheduled 
mission  into  separate  groups.  All  the  assets  of  the  previously 
scheduled  mission  must  be  kept  together,  whether  or  not  their 
target  changes. 

4.  Combine  the  assets  of  two  or  more  previously  scheduled  missions 
and  redirect  them  to  a  different  target. 

The  right  choice  from  among  these  options  must  be  chosen  for  each 
target  in  less  than  three  minutes  of  the  emergence  of  a  TST.  Otherwise,  friendly 
forces  can  be  endangered. 

If  there  are  not  enough  available  assets  to  strike  every  target,  there  may 
be  some  targets  left  unstruck  within  the  vulnerability  window.  Priorities  on  targets 
are  used  to  guide  which  ones  are  most  important  to  strike.  In  solving  this 
problem  the  objectives  are  to: 

1 .  Maximize  achievement  of  target  destruction  goals  (weighted  by  target 

priorities). 

2.  Minimize  attrition  risk  to  employed  assets. 

3.  Disrupt  the  current  ATO  as  little  as  possible. 


4.  Minimize  the  distance  traveled  on  the  newly  assigned  missions. 
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These  objectives  are  combined  into  one  objective  function  with  weights  to 
insure  that  the  first  objective  takes  precedence  over  the  second,  the  second  over 
the  third,  etc. 

The  amount  of  target  destruction  achieved  from  the  assignment  of  assets 
to  targets  depends  on  known  probabilities  of  kill  for  each  ordnance  configuration 
against  each  target  type.  The  attrition  risk  depends  on  known  survival 
probabilities  of  each  aircraft  type  against  each  air  defense  (AD)  type,  and  if 
suppression  of  enemy  air  defense  (SEAD)  is  used,  it  also  depends  on  known 
suppression  probabilities  against  the  AD. 

C.  SUPPRESSION  OF  ENEMY  AIR  DEFENSE 

1.  SEAD  Increases  the  Number  of  Strike  Options 

In  contrast  to  Major  Davi’s  thesis,  the  model  developed  and  tested  here 
explicitly  considers  SEAD.  SEAD  assets,  such  as  the  EA-6B  Prowler  aircraft,  are 
employed  to  neutralize  the  effectiveness  of  the  enemy’s  AD  threats,  such  as  the 
SA-6  surface-to-air  missile  system.  SEAD  does  not  destroy  these  threats, 
instead  it  uses  electronic  means  to  render  the  AD’s  targeting  systems  ineffective. 

A  key  idea  of  Davi’s  modeling  is  that  candidate  air  strike  packages  are 
generated  and  evaluated  against  possible  targets  prior  to  optimizing  for 
recommended  assignments.  An  air  strike  package  is  a  collection  of  aircraft  and 
their  associated  ordnance  loadouts,  launched  from  a  common  airbase  (or 
carrier),  which  can  be  assigned  to  a  target.  The  feature  of  enumerating  the 
possible  package  assignments  prior  to  the  optimization  allows  the  nonlinear 
calculation  of  probabilities  of  kill  and  attrition  to  be  handled  efficiently  within  a 
linear  optimization  model.  It  also  makes  it  easy  to  enforce  constraints  on 
allowable  assignments.  [See  Davi,  page  20,  for  details.] 

The  addition  of  SEAD  to  the  modeling  of  the  problem  allows  for  the 
admissibility  of  more  air  strike  packages  than  previously  allowed.  In  Davi’s  work, 
the  effectiveness  of  a  strike  package  was  computed  based  on  what  the  survival 
probabilities  for  the  aircraft  in  the  package  would  be  if  they  were  engaged  by  the 
target’s  air  defense  threat.  In  some  cases,  the  number  of  aircraft  actually 
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reaching  the  target  in  the  absence  of  SEAD  would  be  diminished  to  the  point  that 
the  package  is  ineffective  and  therefore  not  considered  any  further.  Assigning  a 
SEAD  asset  to  accompany  a  package  increases  the  survivability  of  the  strike 
aircraft,  thereby  increasing  the  number  of  options  that  can  be  considered  to  strike 
the  target. 

2.  “Man-to-Man”  vs.  “Zone”  Defense 

The  employment  of  SEAD  assumes  that  the  SEAD  asset  can  support  only 
one  mission  at  a  time.  The  basketball  analogy  of  “man-to-man  defense”  is  used 
to  describe  this  assumption.  “Zone  defense”  would  allow  a  SEAD  asset  to 
suppress  threats  for  more  than  one  mission,  but  this  is  not  considered  here. 
“Man-to-man”  is  consistent  with  current  tactics.  It  is  not  assumed,  however,  that 
SEAD  assets  must  remain  with  their  supported  assets  after  the  strike.  The 
SEAD  may  begin  supporting  another  strike  package  before  the  strike  aircraft  in 
the  previous  package  return  to  their  base(s). 

3.  Implications  and  Assumptions  of  SEAD  Employment 

Due  to  the  “man-to-man”  assumption,  a  SEAD  asset  can  be  pre-defined 
as  part  of  a  strike  package.  The  effectiveness  and  attrition  probabilities  of  the 
strike  packages  can  still  be  computed  prior  to  optimization,  preserving  the 
computational  advantages  enjoyed  by  Davi’s  approach.  The  formulas  used  for 
these  probabilities  are  given  in  Section  D.  They  assume  air  superiority  during  the 
conduct  of  TST  strike  operations.  Air  superiority  does  not  imply  the  absence  of 
threat,  but  it  does  assume  that  aircraft  striking  a  TST  will  not  be  countered  with 
an  overwhelming  number  of  surface-to-air  missiles. 

The  probability  calculations  assume  that  each  threat  system  will  fire  one 
missile  at  each  strike  aircraft.  This  assumption  is  based  on  air  superiority  during 
offensive  air  support  operations,  and  the  existence  of  a  degraded  enemy  air 
defense  system.  If  the  assumption  were  wrong  and  missiles  outnumbered 
aircraft,  the  aircraft  survival  estimates  would  probably  be  optimistic. 

The  probabilities  of  survival  and  suppression  included  in  the  model  runs 
reported  in  this  thesis  are  unclassified  and  were  provided  by  SPAWAR. 
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D.  OPTIMIZATION  MODEL 

This  section  provides  the  formulation  of  the  optimization  model  developed 
and  tested  in  this  thesis. 

1.  Strike  Package  Eligibility 

In  Major  Davi’s  models,  any  combination  of  aircraft  and  ordnance  that 
pass  muster  with  the  judgment  and  experience  of  the  strike  planners  can  be 
considered  for  inclusion  as  a  potentially  recommended  air  strike  package.  For 
the  time-critical  situation  considered  in  this  thesis,  there  is  far  less  freedom  of 
choice.  The  aircraft  and  ordnance  configurations  as  they  appear  on  the  current 
ATO  are  the  only  assets  that  can  be  considered.  The  only  types  of  strike 
packages  allowed  are: 

1 .  Air  strike  packages  currently  on  alert  status. 

2.  Air  strike  packages  currently  assigned  to  missions  within  the  vulnerability 

window. 

3.  The  union  of  the  assets  in  two  or  more  air  strike  packages  of  the  first  two 

types. 

4.  Air  strike  packages  of  the  first  three  types  with  the  addition  of  SEAD. 

There  is  not  enough  time  to  break  up  the  assets  in  a  current  mission  into 
more  than  one  air  strike  package.  Nor  is  there  enough  time  to  reconfigure  an 
available  aircraft’s  weapons  loadout,  even  if  it  takes  a  reconfiguration  to  achieve 
maximum  effectiveness  against  the  TST. 

The  time  horizon  for  the  problem  treated  in  this  thesis  is  too  brief  to  allow 
using  strike  aircraft  against  more  than  one  target.  This  is  in  contrast  to  the 
dynamic  model  in  Chapter  III  of  Major  Davi’s  thesis.  His  dynamic  model  explicitly 
accounts  for  the  time  required  to  fly  multiple  missions,  including  the  time  between 
missions  for  reloading  ordnance,  refueling  and  making  other  necessary 
preparations. 

2.  Indices  and  Sets 

The  following  section  identifies  all  indices  and  sets  used  in  the  formulation 
of  the  optimization  model. 
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j  Targets  requiring  assignment  (including  one  or  more  TST,  and  the 
targets  currently  scheduled  on  the  ATO  during  the  vulnerability 
window),  e.g.,  {SS-21,  T-72,  Logistics  Site,...} 

m  Missions  currently  scheduled  on  the  ATO  to  either  strike  targets  or 
stand  on  alert  during  the  vulnerability  window,  {  M11 11,  Ml  112,...} 

n  Available  air  strike  packages,  as  explained  in  Section  D.1 
{N1,N2,...} 

The  resources  to  be  allocated  in  this  optimization  model  are  the  assets 
already  assigned  to  the  missions  indexed  by  m.  Some  of  these  assets  may 
continue  to  fly  to  their  intended  targets,  while  others  may  be  re-tasked  to  a 
different  target. 

There  are  some  situations  in  which  two  targets  on  the  ATO  are  logically 
connected.  For  example,  an  SS-21  may  be  one  target  defended  by  another 
target,  say,  an  SA-6.  The  destruction  of  the  SS-21  cannot  be  achieved  without 
destroying  the  SA-6.  This  relationship  is  identified  in  a  pre-processing  step  and 
treated  in  the  model  by  combining  the  two  targets  into  one. 

The  following  input  data  define  important  subsets  of  the  index  sets.  They 
specify  which  targets  may  be  struck  by  which  packages  and  which  missions’ 
assets  comprise  which  package. 

rij(n,j)  =  1  if  target  j  can  be  assigned  to  air  strike  package  n;  0  otherwise. 
This  depends  on  whether  the  package  has  a  sufficiently  high 
probability  of  achieving  the  required  destruction,  and  whether  it  can 
do  so  inside  the  vulnerability  window  and  within  an  acceptable  level 
of  attrition  risk. 

nm(n,m)  =  1  if  air  strike  package  n  contains  the  assets  of  mission  m;  0 
otherwise. 
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3.  Parameters 

The  data  required  to  execute  the  optimization  model  are  the  penalty  costs 
associated  with  assigning  a  particular  strike  package  to  a  target,  and  the  penalty 
cost  for  leaving  it  unstruck. 

pen_nogo(j)  =  penalty  for  not  striking  target  j 

pen_attrition(n,j)  =  penalty  for  probable  aircraft  loss  if  strike  package  n  is 
assigned  to  target] 

pen_change(n,j)  =  penalty  for  causing  changes  in  assets’  target 
assignments  if  strike  package  n  is  assigned  to  target] 

pen_distance(n,])  =  penalty  for  aircraft  travel  distance  if  strike  package  n 
is  assigned  to  target]. 

The  possibility  exists  that  at  the  time  a  TST  emerges,  all  assets  may  be 
assigned  targets  on  the  ATO.  Although  the  ATO  allows  for  aircraft  to  be  in  alert 
status  without  assignment  to  a  dedicated  target,  this  situation  cannot  always  be 
relied  upon.  Even  if  there  are  aircraft  in  alert  status,  they  may  not  have  the 
necessary  ordnance  configuration  to  successfully  engage  the  TST.  In  this  case 
assets  will  be  diverted  to  the  TST,  and  some  other  target  may  go  unstruck.  The 
model  seeks  in  this  case  to  choose  a  low-priority  target  for  non-assignment.  If  a 
previously  assigned  target  becomes  unassigned  after  re-tasking  for  the  TST, 
then  the  unassigned  target  will  be  considered  in  a  future  run  of  the  model. 

These  penalties  are  derived  from  raw  problem  data  and  the  mission 
planner’s  ]udgment,  as  described  in  Section  D.7. 

4.  Decision  Variabies 

STRIKE(n,])  =  1  if  air  strike  package  n  is  assigned  to  target],  0  otherwise 

NOGO(])  =  1  if  no  strike  package  is  assigned  to  target],  0  otherwise 

IDLE(m)  =  1  if  mission  m  is  not  used  in  any  of  the  assigned  packages,  0 
otherwise 

The  STRIKE  variables  need  to  be  treated  as  binary  decision  variables. 

The  other  two  types  of  variables  can  be  treated  as  continuous,  since  the 
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constraints  will  force  them  to  take  on  binary  values  when  STRIKE(n,j)  are  all 
binary.  The  variable  IDLE(m)  accounts  for  the  possibility  that  the  assets 
associated  with  a  mission  will  not  be  needed  for  any  of  the  current  target 
assignments.  The  commander  may  choose  to  put  these  assets  into  the  alert 
status. 

5.  Formulation 

Minimize 

^  [pen_attrition(nJ)  +  pen_change(nj)+  pen_distance(nj)]  *  STRIKE(nJ) 

(nj) :  nj(n,j)=1 

+  ^pen_nogo(j)  *  NOGO(j) 

j 

s.t. 

^  STRIKE(nJ)  +  NOGO(j)  =  1 ,  Vj 

n  :  nj(n,j)=1 

^  nm(n,m)*STRIKE(n J)  +  IDLE(m)  =  1 ,  Vm 

(nj) :  nj(n,j)=1 


The  first  set  of  constraints  ensures  that  each  target  is  struck  by  at  most 
one  feasible,  available  strike  package,  or  the  target  is  left  unstruck  due  to 
insufficient  assets.  Because  the  time  horizon  is  so  short,  the  second  set  of 
constraints  ensures  that  the  assets  in  each  mission  are  employed  at  most  once. 

6.  Given  Data  from  which  Parameters  are  Derived 

Three  additional  indices  are  needed  to  specify  the  raw  input  data.  They 
are  not  needed  for  the  optimization  model  formulation  because  they  are  used 
only  in  calculations  of  parameters  prior  to  the  optimization. 

a  Aircraft  types  {FA1 8,  EA6B, . . .} 

r  Threat  types  {SA6,SA8,  ...} 

w  Weapon  types  currently  carried  by  missions  on  the  ATO,  {MK83, 
MK84,..} 
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The  raw  data  for  an  instance  of  the  problem  consist  of  the  following 
parameters: 

P_REQUIRED(j)  Commander’s  required  probability  of  success  for 
target  j 

SSP_KILL(wJ)  Single-shot  probability  of  kill  for  weapon  type  w 
against  target] 

NUMAC(m)  Number  of  aircraft  in  mission  m  on  the  ATO 

CONFIG(m,w)  Number  of  type  w  weapons  loaded  on  each  aircraft  of 

mission  m  on  the  ATO 

threat(j)  =  r  such  that  target  j  is  known  to  be  defended  by  a 

threat  system  of  type  r 

SSP_SUPPRESS(r,  w)  Single-shot  probability  of  suppression  of  a  threat 

of  type  r  by  a  weapon  of  type  w 

actype(m)  =  a  such  that  mission  m  employs  aircraft  type  a  in  the 

current  ATO 

SSP_SURVIVAL(r,a)  Single-shot  probability  of  survival  of  an  aircraft  of 
type  a  when  challenging  a  threat  of  type  r 

SSP_KILL(wJ)  Single-shot  probability  of  kill  for  a  weapon  of  type  w 
against  target] 

PRIORITY(])  Commander’s  priority  for  target  ].  (Smaller  numbers 

mean  higher  priority) 

PREV_ASSIGN(mJ)  =  1  if  mission  m  is  assigned  to  target  ]  on  the  ATO, 

0  otherwise.  LAT(m),  current  latitude  of  the  strike 
aircraft  on  mission  m 

LON(m)  Current  longitude  of  the  strike  aircraft  on  mission  m. 

PRIORITY_WEIGHT,  ATTRITION_WEIGHT,  CHANGE_WEIGHT, 

DISTANCE_WEIGHT  Weighting  factors  for  ob]ective  function  terms 
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7. 


Derivation  of  Probabiiities  and  Penaities 


Given  the  data  provided  in  the  previous  section  the  following  probabilities 
and  penalties  are  derived  prior  to  optimization. 

a.  Probability  of  Suppression 

Probability  of  suppression  is  defined  as  the  probability  of 
successfully  disabling  the  threat  using  package  SEAD  assets.  It  is  calculated 
from  the  single  shot  probabilities  of  suppression  of  each  SEAD  weapon  in  the 
package.  It  uses  an  intermediate  calculation  of  the  weapon  count  for  strike 
package  n. 

NUMWEAPON(n,w)  =  ^  nm(n,m)  *  NUMAC(m)  *  CONFIG(n,m) 

m 

P_SUPPRESS(nJ)  =  1-  SSP_SUPPRESS(threat(j))”^''^°^<"'*>  V  nj 

W 

b.  Probability  of  A  ttrition 

Probability  of  attrition  is  defined  as  the  probability  of  losing  at  least 
one  aircraft  from  a  package  during  a  strike  against  a  target.  If  there  is  no  threat 
associated  with  a  target,  the  probability  of  attrition  is  zero.  If  there  is  a  threat 
associated  with  a  target,  the  probability  is  calculated  as  the  cumulative  probability 
of  SEAD  assets  failing  to  suppress  the  threat  and  the  probability  that  at  least  one 
aircraft  in  the  package  fails  to  survive  the  unsuppressed  threat. 

P_ATTRITION(n,j)  =  (1  -  P_SUPPRESS(n,j))  *  (1  -  P_SURVIVAL(n,j)) 

c.  Probability  of  Survival 

Probability  of  survival  of  all  aircraft  in  a  package  is  calculated  from 
the  single  shot  probabilities  of  survival  for  each  aircraft  in  the  package  against 
the  given  threat. 

P_SURVIVAL(n,j)  =  1-  Y[  ('I  '  SSP_SURVIVAL(threata),  actype(m))  )) 

m  :  nm(n,m)=1 

This  method  assumes  a  threat  has  the  ability  to  take  a  single  shot 
at  each  aircraft  in  the  package.  This  is  a  worst-case  scenario  that  can  be  used 
when  there  is  incomplete  information  about  how  many  weapons  a  threat  may 
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have  at-the-ready  or  in  inventory  and  how  quickly  those  weapons  can  be 
reloaded. 

d.  Probability  of  Kill 

Probability  of  kill  is  defined  as  the  probability  an  air  strike  package 
can  destroy  a  target  ignoring  the  interference  of  any  threats.  It  is  calculated  from 
single  shot  probabilities  of  all  weapons  in  the  package. 

P_KILL(nJ)=  1-]^(1-SSP_KILL(W,j))  NUMWEAPON(n,w)  ^ 

w 

e.  Probability  of  Success 

Probability  of  success  is  defined  as  the  probability  of  destroying  a 
target  without  loss  of  aircraft.  It  can  be  computed  from  the  probability  of  attrition 
and  probability  of  kill. 

P_SUCCESS(nJ)  =  (1  -  P_ATTRITION(nJ))  *  P_KILL(nJ)  V  nj 

This  computed  probability  is  compared  to  the  required  probability  of 
success  for  target  j  to  decide  if  assigning  package  n  to  target]  should  become  an 
allowable  option  in  the  optimization.. 

f  Penalty  for  Not  Striking  a  Target 

The  penalty  for  not  striking  a  target  is  calculated  by  weighting  a 
priority  value  that  increases  with  higher  priority  targets.  A  priority  exponent 
greater  than  1  (usually  2)  is  used  to  accentuate  the  importance  of  top  priority 
targets  relative  to  lower  priority  targets. 

PEN_NOGO(j)  =  PRIORITY_WEIGHT  *  (  1  /  PRIORITY^  )  pR'oRity.exponent 

g.  Penalty  for  Losing  an  Aircraft 

The  penalty  for  losing  an  aircraft  in  a  strike  is  calculated  by 
weighting  the  probability  of  attrition. 

PEN_ATTRITION(nJ)  =  ATTRITION_WEIGHT  *  P_ATTRITION(nJ) 
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h.  Persistence  Penalty 

The  persistence  penalties  for  changing  targeting  assignments  of 
strike  aircraft  are  calculated  by  weighting  the  number  of  changes  or  “phone  calls” 
required  to  implement  the  solution. 

PEN_CHANGE(n,j)  =  CHANGE_W EIGHT  * 

^  (1  -  PREV_ASSIGN(n,j))  V  nj 

m  :  nm(n,m)=1 

/.  Distance  Penalty 

The  distance  penalty  is  calculated  by  weighting  the  nautical  miles 
required  for  all  aircraft  in  a  package  to  reach  a  target. 

PEN_DISTANCE(n,j)  =  DISTANCE_WEIGHT  * 

^  NUMAC(m)*  DISTANCE(m,j)  V  n,j 

m  :  nm(n,m)=1 

Distance  can  be  calculated  as  the  great  circle  distance  between  the  two 
locations  or  obtained  from  mapping  software  if  there  are  limitations  on  direct 
routes.  In  the  case  of  the  tactical  evaluation  discussed  in  Chapter  IV,  distances 
were  ignored  because  they  were  judged  to  be  too  small  to  have  a  realistic  impact 
on  decisions. 

E.  GRAPHICAL  USER  INTERFACE 

With  the  help  of  the  author,  Mr.  Anton  Rowe,  a  research  associate  in  the 
NPS  Operations  Research  Department,  we  designed  and  built  a  spreadsheet- 
based  graphical  user  interface  (GUI)  for  the  optimization  model  described  above. 
The  model  is  extremely  easy  to  use  with  this  GUI. 

The  principle  interactive  displays  of  the  GUI  are  illustrated  in  Figures  5-7. 
Figure  5  shows  the  Target  Screen,  in  which  each  target  is  listed  with  its  priority, 
air  defense  threat  (if  any)  and  required  probability  of  successful  attack.  There  is 
an  indicator  field  that  specifies  which  targets  are  included  in  the  current  run  of  the 
model.  In  the  example  of  Figure  5,  the  SS-21  on  the  bottom  row  is  a  TST. 
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Targets 


ProbabBty  F 

Include  Target  Priority  Type  Threat  of  Success 
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5 
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Figure  5  Target  Screen.  A  screen  shot  provided  by  the  model  interface.  It 
depicts  all  pre-planned  and  designated  TSTs  indicated  by  the  far  left 
column.  The  current  threat  associated  with  each  target  and  the  required 
probability  of  success  are  indicated  in  far  right  columns. 

Figure  6  displays  the  Mission  Screen,  which  represents  the  current  ATO. 
The  missions  selected  on  this  screen  are  the  ones  whose  assets  are  to  be 
allocated  in  the  optimization.  For  each  mission,  there  is  a  call  sign,  times  of 
departure  and  return,  number  and  type  of  aircraft  in  the  mission,  weapon 
configuration,  and  the  current  ATO’s  target  assignment.  Pressing  the 
“Recommend”  button  causes  the  optimization  model  to  be  generated  and  solved, 
with  the  resulting  target  assignments  displayed  in  the  right-most  column.  It 
generally  takes  about  three  seconds  on  a  personal  computer  to  obtain  an  optimal 
solution  after  about  a  minute  or  two  of  user  input  time.  After  the  optimization 
model  is  solved,  the  Target  Screen  is  updated  as  in  Figure  7  to  show  the 
important  results.  These  include  the  achieved  probability  of  successful  attack  on 
each  target  and  the  probability  of  attrition  for  the  assigned  strike  packages. 
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Missions 


Recommend 


Clear 


Indude 

Mission 

C/S 

FID 

Ere 

# 

Aircraft 

Configuration 

Assignment 

Recommend 

X 

6111 

Razor  11 

15:51 

17:07 

2 

AV8B 

[2] CBU58 

TGT  SA3 

SS21 

6113 

Razor  13 

2 

AV8B 

[4]  MK83 

TGT  SA3 

X 

6141 

Razor  41 

15:51 

17:07 

2 

AV8B 

[2] CBU58 

TGT  2 

X 

6143 

Razor  43 

15:51 

17:07 

2 

AV8B 

[4]  MK83 

TGT  2 

TGT  2 

X 

6115 

Razor  15 

15:51 

17:07 

2 

AV8B 

[4]  MK83 

TGT  3 

TGT  3 

X 

6145 

Razor  45 

15:51 

17:07 

2 

AV8B 

[4]  MK83 

SLA  15 

TGTSA3 

X 

6147 

Razor  47 

15:51 

17:07 

2 

AV8B 

[6]  MK83 

SLA  15 

X 

6161 

Razor  61 

15:51 

17:07 

2 

AV8B 

[4]  MK83 

KB  AA80 

KA  AA80 

X 

6101 

Storm  01 

16:19 

19:09 

2 

EA6B 

[1]  POD  [1]  HARM 

TGT  SA3 

TGT  SA3 

6103 

Storm  03 

16:19 

19:09 

1 

EA6B 

[1]  POD  [1]  HARM 

TGT  2 

6104 

Storm  04 

16:19 

19:09 

1 

EA6B 

[1]  POD  [1]  HARM 

SLA  30 

6131 

Latch  31 

16:34 

17:54 

2 

F18D 

[4]  MK83 

TGTl 

6133 

Latch  33 

16:34 

17:54 

2 

F18D 

[4]  MK83 

TGTl 

6135 

Latch  35 

16:34 

17:54 

2 

F18D 

[4]  MK83 

TGT  3 

6137 

Latch  37 

17:04 

18:24 

2 

F18D 

[4]  MK83 

KBAA80 

6175 

Latch  75 

17:49 

19:09 

2 

F18D 

[4]  MK83 

KB  AA80 

6163 

Razor  63 

18:07 

19:21 

2 

AV8B 

[4]  MK82 

SLA  15 

6165 

Razor  65 

18:22 

19:36 

2 

AV8B 
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Figure  6  Mission  Screen.  The  output  from  the  interface  of  the  model 
contains  all  the  original  ATO  information.  The  left-most  column  indicates 
the  available  resources  that  will  be  the  input  to  the  model  and  reflect  the 
assets  currently  available  during  the  target’s  vulnerabiiity  window.  In  the 
top  right  hand  corner  of  the  display  the  “Recommend”  Button  can  be  seen. 
Selecting  this  button  will  run  the  optimization  model  and  place  the  results 
in  the  far  right  column.  The  recommend  column  identifies  all  assignments 
to  be  made. 
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Figure  7  Target  Worksheet.  The  target  worksheet  depicts  each  target 
included  in  the  input  to  the  model.  For  each  target  a  priority  and  desired 
probability  of  success  is  provided.  The  column  headed  by  “Probability 
Achieved”  indicates  for  each  asset  assigned  on  the  mission  worksheet  to  a 
target,  the  desired  probability  of  kill  has  been  met. 

E.  Tactical  Example 

The  problem  illustrated  in  Figures  5-7  is  an  actual  example  from  testing 
the  model  in  live  exercises  at  Marine  Corps  Air  Station  Yuma.  The  testing  is 
described  in  detail  in  Chapters  IV  and  V.  As  can  be  seen  from  Figure  7,  the 
recommended  solution  is  tactically  accurate  in  the  sense  that  every  target  is 
struck  with  the  required  probability  of  success.  In  this  case,  the  model  redirected 
a  surprisingly  large  number  of  missions  to  different  targets.  This  is  in  contrast  to 
the  solution  obtained  manually  by  the  control  group  in  the  testing.  Their  solution 
involved  no  redirection  because  it  engaged  the  TST  with  Mission  6145,  which 
was  on  strip  launch  alert.  The  explanation  of  this  initially  disappointing  result 
from  the  model  was  that  it  turned  out  that  one  of  the  current  taskings  on  the  ATO 
did  not  achieve  the  required  probability  of  success.  In  this  way,  the  optimization 
model  identified  a  potentially  dangerous  mistake  in  the  current  ATO.  This  was  an 
unintended  benefit  of  using  the  new  model.  The  next  chapter  describes  the  test 

plan  for  evaluating  the  new  model  in  a  live  tactical  exercise. 
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IV.  TEST  AND  EVALUATION 


A.  INTRODUCTION 

Test  and  Evaluation  is  the  “process  by  which  a  system  or  components  are 
evaluated  or  compared  against  requirements  and  specifications  through  testing 
or  experimentation.  The  results  are  evaluated  to  assess  progress  of  design, 
performance,  and  supportability.”  [Test  and  Evaluation  Lecture  Notes,  T.H. 
Hoivik,  2004]  The  test  and  evaluation  of  the  decision  aids  developed  by 
SPAWAR  and  NPS  addresses  the  second  objective  of  this  thesis.  This  chapter 
explains  the  methodology  developed  and  criterion  used  to  support  the  evaluation 
of  both  the  SPAWAR  and  NPS  Decision  Aids  during  a  realistic  tactical 
environment. 

One  problem  in  evaluating  a  new  initiative  or  concept  is  determining 
precisely  what  the  new  initiative  or  concept  is  supposed  to  accomplish,  matching 
the  appropriate  measures  to  determine  if  accomplishment  has  occurred,  and 
identifying  criteria  for  measurement  of  its  overall  worth.  This  thesis  will  utilize  the 
Dendritic  methodology'll  for  deriving  the  decision  aids  critical  operational  issues, 
measures  of  effectiveness,  measures  of  performance,  data  requirements,  criteria 
for  evaluation,  and  estimating  the  overall  worth  of  these  decision  aid’s. 

Proper  operational  test  designs  are  critical  to  identification  and  evaluation 
of  factors  that  influence  the  new  concepts  worth.  Otherwise,  many  so-called 
“tests”  devolve  into  feasibility  demonstrations  and  the  resulting  data  and 
outcomes  may  not  be  credible  or  applicable  for  extension  into  real  world 
situations.  Unfortunately,  the  highly  controlled  and  structured  test  methods  used 
for  accepting  or  rejecting  hypothesis  with  a  very  high  degree  of  certainty  in 
confirmatory  analysis  do  not  apply  to  complex  operational  tests.  Simply,  there 
are  too  many  variables  to  control  and  realism  would  be  sacrificed.  However, 
using  select  principles  of  test  and  experimental  design  in  exploratory  analysis, 
important  insights  into  factors  that  effect  or  influence  the  worth  of  initiative  or 

Fundamentals  of  Military  Experiment  Planning,  Design  and  Analysis,  by  Thomas  H.  Hoivik, 
Naval  Postgraduate  School,  2002 


33 


concept  can  be  identified  even  though  statistical  significance  may  not  be 

calculated, 

B.  DENDRITIC  METHODOLOGY 

The  dendritic  methodology  is  a  process  for  deriving  a  system  or  concept’s 
Critical  Operational  Issues  (COI),  Measures  of  Effectiveness  (MOE),  Measures  of 
Performance  (MOP),  and  Data  Requirements  (DR)  for  the  purposes  of 
operational  test  and  evaluation  or  experimentation.  Once  data  requirements  are 
established,  factors  and  conditions  needed  for  appropriate  evaluation  can  be 
identified  and  appropriate  operational  test  scenarios  or  profiles  can  be  generated 
for  inclusion  in  the  overall  test. 

The  goal  of  the  dendritic  method  is  to  identify  relevant  Measures  of 
Performance  (MOP)  needed  to  analyze  and  evaluate  performance  (operational 
usefulness)  of  a  specific  concept,  or  initiative  and  to  answer  or  resolve  their 
associated  operational  issues.  Since  MOPs  are  usually  rates,  ratios,  percents,  or 
some  quantitatively  or  qualitatively  generated  number,  each  MOP  will  define  its 
required  data  elements. 

The  dendritic  method  starts  by  identifying  all  the  functions  and  tasks  that  a 
specific  decision  aid  is  to  perform.  The  most  important  decision  aid  functions  and 
tasks  will  be  transformed  into  operational  issues  (questions)  that  need  to  be 
evaluated  (answered)  during  the  experiment.  The  MOE  for  a  decision  aid 
generally  relate  to  the  capabilities  desired  for  each  function  or  task.  Once  MOEs 
have  been  identified,  then  appropriate  quantitative  or  qualitative  MOP  (rates, 
ratios,  percents,  etc)  can  be  developed  to  evaluate  the  effectiveness  of  each 
decision  aid  capability.  It  should  be  noted  that  several  MOPs  may  be  needed  for 
each  MOE  and  several  MOEs  are  usually  needed  to  fully  encompass  the 
evaluation  of  an  experimental  issue.  With  required  and  appropriate  MOEs,  mini¬ 
scenarios  or  run  profiles,  which  are  sets  of  factors  and  conditions,  can  be 
constructed  to  generate  the  required  data.  The  test  director  while  maintaining 
operational  realism  can  insert  these  mini-scenarios  or  run  profiles  repeatedly. 
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and  at  random,  to  foster  efficient  and  credible  data  generation  and  comparative 
analysis.  Therefore,  even  though  the  overall  operational  test  may  be  complex 
and  unstructured,  the  insertion  of  mini-scenarios  or  run  profiles  with  appropriate 
systematic  variation  of  the  relevant  factors  and  conditions  will  allow  for  enhanced 
analysis  and  evaluation.  This  will  foster  meaningful  insight  into,  or  identification 
of,  those  factors  that  influence  the  mission  success  or  failure  of  a  concept, 
process  or  decision  aids,  "is 
C.  CRITICAL  OPERATIONAL  ISSUES  (COI) 

First  and  foremost,  to  develop  an  effective  evaluation  the  critical 
operational  issues  of  the  decision  aids  need  to  be  determined.  The  dendritic 
method  was  used  to  develop  the  test  plan  used  to  evaluate  the  decision  aids. 
The  following  is  a  summary  of  the  process  used  to  derive  the  COI’s,  MOE’s,  and 
MOP’S  derived  to  evaluate  the  decision  aids.  The  first  step  in  the  dendritic 
method  is  to  determine  the  decision  aid’s  primary  functions.  The  major 
capabilities  desired  for  each  function  are  then  enumerated  as  shown  in  Table  1. 
These  capabilities  may  then  be  subdivided  into  more  specific  capabilities  if 
desired.  This  initial  breakdown  of  functions  and  capabilities  provides  the  basis 
for  determining  critical  operational  issues  and  system  measures  of  effectiveness. 
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Function 

Capability 

Target  Selection 

Ensure  target  precedence  is  maintained  during  asset  assignments 

Asset  Avaiiability 

Assignments  are  oniy  comprised  of  assets  that  are  abie  to  strike  the  asset  within  the 

prescribed  time. 

Target-Asset  Pairing 

Method  of  attack  that  is  seiected  must  be  abie  to  achieve  the  prescribed  destruction 

criteria. 

Mission  Risk  Assessment 

1 .  Abie  to  asses  risk  to  muitipie  aircraft  piatforms  from  various  surface-to-air  threats 

2.  Assigns  SEAD  support  when  it  is  required. 

Persistence 

Limits  the  number  of  disruptions  to  the  current  ATO. 

Timeliness 

Determines  soiution  in  enough  time  to  take  effective  action. 

Options 

Provides  muitipie  courses  of  action. 

Tactical  Accuracy 

Provides  soiutions  that  are  acceptabie  in  a  iive  tacticai  environment. 

Interoperability 

Can  receive  Air  Tasking  Orders  from  current  systems  empioyed. 

Reliability 

Perform  without  interruption  for  24  hours. 

Table  1.  Decision  Aid  Functions  and  Capabilities. 


COIs,  are  usually  the  critical  functions  a  system  must  perform. "i®  COI’s 
provide  the  focus  and  direction  for  our  test  and  evaluation.  COI’s  are  usually 
defined  as  questions  to  be  resolved  during  the  operational  test.  The  COI’s  used 
in  the  evaluation  of  our  decision  aids  are; 

1.  COM.  Target  Selection.  Do  the  decision  aids  properly  conduct  target 

prioritization  of  targets  to  ensure  mission  accomplishment  of  higher 

priority  targets  before  lower  priority  targets? 

2.  COI  2.  Asset  Availability.  Do  the  decision  aids  properly  recognize  the 

assets  that  are  available  for  a  particular  target? 

3.  COI  3.  Target  Asset  Pairing.  Do  the  decision  aids  properly  recommend 

targets  that  have  the  ability  to  destroy  the  target? 

4.  0014.  Mission  Risk  Assessment.  Do  the  decision  aids  accurately 

assess  the  risk  of  its  proposed  assignments? 

5.  COI  5.  Persistence.  Do  the  decision  aids  minimize  the  number  of 

changes  to  the  ATO  to  achieve  mission  accomplishment? 

6.  COI  6.  Timeliness.  Do  the  decision  aids  provide  a  proposed  solution  fast 

Fundamentals  for  Military  Experiment  Planning,  Design  and  Analysis,  by  Thomas  H. 
Hoivik,  Naval  Postgraduate  School  2002 


36 


enough  to  be  effective  when  compared  to  current  methods? 

7.  COI  7.  Options.  Do  the  decision  aids  provide  the  decision  maker  with 

multiple  alternative  solutions  for  mission  re-tasking  when  available? 

8.  COI  8.  Tactical  Accuracy.  Do  the  decision  aids  output  provide  tactically 

acceptable  assignments? 

9.  COI  9.  Interoperability.  Do  the  decision  aids  operate  correctly  with  the 

current  C4I  architecture? 

10.  CO1 10.  Software  Reliability.  Can  the  decision  aids  operate  continuously 

without  interruption  or  failure  for  a  prolonged  period  of  time? 

1 1 .  CO1 1 1 .  Human  Factors.  Can  the  typical  user  enter  inputs  and 

understand  the  outputs  in  an  efficient,  intuitive  manner  with  minimal 

training? 

The  following  sections  will  describe  each  COI  in  detail,  and  justify  their 
inclusion  in  the  evaluation. 

1.  Do  the  decision  aids  properly  conduct  target  prioritization  of 
targets  to  ensure  mission  accomplishment  of  higher  priority 
targets  before  lower  priority  targets? 

The  senior  commander  establishes  the  priority  of  all  targets  on  the 
battlefield  and  deviations  from  this  can  only  occur  with  the  consent  of  that 
commander.  Targets  are  prioritized  numerically  beginning  with  one,  the  highest 
priority  target.  Given  a  situation  where  all  assets  are  currently  assigned,  the 
potential  for  a  target  not  being  struck  does  exist.  Each  decision  aid  must  be 
able  to  accurately  assign  assets  so  that  the  higher  priority  targets  are  paired  with 
assets.  Time  Sensitive  Targets  are  the  highest  priority  targets  on  the  battlefield; 
successful  prosecution  of  these  targets  supercede  all  other  priorities  on  the 
battlefield. 

2.  Do  the  decision  aids  properly  recognize  the  assets  that  are 
available  for  a  particular  target? 

Available  assets  are  those  that  have  not  been  cancelled  on  the  ATO  for 
mechanical  failures  and  those  that  have  not  been  committed  to  their  original 
target  and  therefore  are  available  for  possible  diversion.  Current  asset 
availability  can  be  determined  either  manually  or  through  automated  updates 
received  over  the  tactical  data  network.  In  either  case  each  decision  aid  must  be 
able  to  recognize  changes  in  the  status  of  the  current  ATO.  This  will  allow  each 
decision  aid  to  properly  determine  the  set  of  feasible  solutions.  The 
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determination  of  available  resources  is  a  target  phase  task  that  was  previously 
identified  in  Chapter  II. 

3.  Do  the  decision  aids  properiy  recommend  targets  that  have 
the  abiiity  to  destroy  the  target? 

Each  asset’s  ordnance  effects  against  a  given  target  type  can  be 
computed  by  the  decision  aids  to  estimate  a  level  of  destruction  or  probability  of 
achieving  the  desired  effect.  This  calculation  must  be  equal  to  or  greater  than 
the  destruction  criteria  required  by  the  commander.  Assets  assigned  that 
achieve  the  desired  destruction  criteria  will  be  considered  a  correct  assignment. 
This  COI  supports  the  following  Target  Phase  tasks  of  (1)  develop  options  to 
achieve  desired  effects,  and  (2)  Weaponeer. 

4.  Do  the  decision  aids  accurateiy  assess  the  risk  of  its  proposed 
assignments? 

Given  a  threat  associated  with  a  target,  each  decision  aid  is  expected  to 
compute  a  level  of  risk  to  the  assets  assignments.  With  regards  to  the  decision 
aid,  it  is  not  the  intent  to  discount  or  discard  those  assignment  options  that 
assume  a  high  risk.  It  is,  however,  intended  to  display  the  risk  associated  with 
each  potential  assignment  to  the  commander  in  order  to  make  an  informed 
decision.  Based  on  this  risk  assessment  a  determination  by  the  decision  aid  will 
be  made  to  determine  whether  or  not  destruction  criteria  can  be  met  or  if  BEAD 
assets  are  recommended  to  achieve  desired  effects. 

5.  Do  the  decision  aids  minimize  the  number  of  changes  to  the 
ATO  to  achieve  mission  accompiishment? 

Each  decision  aid  is  expected  to  select  an  asset  for  assignment  that 
causes  the  least  impact  to  the  existing  plan.  In  developing  a  decision  aid  to  be 
employed  in  a  tactical  environment,  the  number  of  re-assignments  to  be  made 
and  the  effort  required  to  implement  the  re-assignment  must  be  considered.  The 
impact  to  the  ATO  is  measured  by  the  number  of  radio  calls  required  to  redirect 
aircraft  to  their  new  assignments.  The  number  of  radio  calls  reflects  the  amount 
of  coordination  required  to  ensure  safe  and  effective  execution  of  the  decision. 
Any  redirection  of  aviation  assets  is  a  disruption;  too  many  disruptions  can  result 
in  an  unsafe  and  eventually  ineffective  execution. 
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6.  Do  the  decision  aids  provide  a  proposed  solution  fast  enough 
to  be  effective  when  compared  to  current  methods? 

The  timeline  needed  to  generate  a  proposed  solution,  decide  on  the 
course  of  action,  and  then  implement  the  decision  is  of  the  utmost  importance.  In 
Chapter  II,  Figure  4,  a  recommended  timeline  for  decision  tools  to  be  effective  in 
support  of  the  TST  process  was  identified.  However,  given  a  tactical 
environment,  appropriate  timeliness  may  be  better  determined  by  experienced 
decision  makers.  To  resolve  this  COI  a  comparative  analysis  will  be  conducted 
to  determine  if  there  is  a  significant  improvement  in  timeliness  over  existing 
manual  methods. 

7.  Do  the  decision  aids  provide  the  decision  maker  with 
multiple  alternative  solutions  for  mission  re-tasking  when 
available? 

There  is  no  standard  among  decision  makers  to  obtain  a  baseline  for  how 
a  proposed  assignment  will  be  perceived  in  regards  to  the  effectiveness, 
assessment  of  risk,  or  impact  to  the  ATO.  For  this  reason  the  number  of  options 
a  decision  aid  provides  allows  for  flexibility  among  decision  makers.  Decision 
makers  may  have  different  experiences  or  willingness  to  accept  increased  risk  to 
achieve  the  desired  end  state.  Therefore,  the  more  options  provided  by  the 
decision  aids  the  more  likely  a  proposed  assignment  will  be  accepted. 

8.  Do  the  decision  aids  outputs  provide  tactically  acceptable 
assignments? 

On  a  case-by-case  basis  the  decision  aids’  proposed  assignments  will  be 
evaluated  for  tactical  acceptance.  The  assessment  of  this  COI  is  qualitative  in 
nature  and  for  any  given  situation  there  can  be  multiple  solutions  that  are 
acceptable.  Not  all  of  them  necessarily  need  to  result  in  the  assignment  of  an 
aviation  asset.  For  this  reason  a  comparative  analysis  will  not  be  conducted  for 
this  COI,  rather  the  consistency  of  the  decision  aids  to  produce  acceptable 
assignments  will  be  evaluated  for  tactical  acceptance  regardless  of  whether  or 
not  they  reflect  the  same  course  of  action  selected  by  the  decision  maker. 
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9.  Do  the  decision  aids  operate  correctiy  with  the  current  C4I 
architecture? 

Each  decision  aid,  to  be  effective,  has  to  either  interact  directly  with 
existing  decision  aids  that  support  the  tactical  data  network  or  at  a  minimum 
recognize  the  data  formats  used  by  the  ATO,  Message  Text  Format  (MTF).  For 
each  of  the  decision  aids  evaluated,  the  ATO  provides  all  the  information  used  in 
developing  a  proposed  solution.  Therefore  the  decision  aid  must  be  able  to  read 
the  ATO  in  its  existing  format,  otherwise  access  to  this  required  information  will 
not  be  possible. 

10.  Can  the  decision  aids  operate  continuousiy  without 
interruption  or  faiiure  for  a  proionged  period  of  time? 

Decision  aid  interruptions  can  occur  for  a  variety  of  reasons,  both  external 
and  internal.  External  factors  such  as  power  interruptions  will  not  be  credited 
against  the  reliability  of  the  decision  aids.  However,  given  a  consistent  operating 
environment,  all  interruptions  in  the  operation  of  the  decision  aids  will  be 
recorded  and  evaluated  with  respect  to  software  reliability. 

11.  Can  the  typical  user  enter  inputs  and  understand  the  outputs 
in  an  efficient,  intuitive  manner  with  minimal  training? 

Given  the  complexity  of  current  command  and  control  centers,  such  as  the 
Tactical  Air  Command  Center  (TACC),  a  variety  of  decision  aids  employed  by 
users  place  a  premium  on  training  to  support  usability  of  automated  decision 
aids.  With  this  in  mind,  any  new  decision  aid  has  to  be  intuitive  with  regards  to 
user  interface.  Operation  of  the  decision  aid  interface  will  be  evaluated 
throughout  the  test.  Input/Output  performance  will  be  evaluated  with  respect  to 
training  requirements  and  personnel  capabilities. 

D.  MEASURES  OF  EFFECTIVENESS,  MEASURES  OF  PERFROMANCE, 

AND  DATA  REQUIREMENTS 

1.  Introduction 

The  evaluation  measures  are  critical  to  the  assessment  of  the  operational 
test.  These  measures  express  the  level  of  success  to  which  the  desired 
outcomes  are  achieved.  Both,  MOE  and  MOP,  will  allow  the  results  of  each  COI 
to  be  quantified  and  analytically  compared  in  order  to  present  a  conclusion. 
Each  COI  can  be  resolved  by  a  single  or  multiple  MOEs  and  MOPs. 
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2.  Measures  of  Effectiveness  (MOE) 

An  MOE,  is  a  measure  which  expresses  the  extent  to  which  the  system 
accomplishes  or  supports  a  mission  or  the  task.  MOE’s  are  designed  to  address 
a  COI.  Each  functional  capability  of  the  decision  aids  will  help  in  identifying  and 
defining  the  MOEs.''^ 

3.  Measures  of  Performance  (MOP) 

MOP,  are  a  quantitative  or  qualitative  measure  of  the  system’s  capabilities 
or  specific  performance  function.  MOP’s  are  usually  in  the  form  of  rates,  ratios, 
or  percents. ''s  Development  of  the  MOPs  using  the  dendritic  methodology  allows 
the  operational  test  and  evaluation  of  outcomes,  either  quantitatively  or 
qualitatively.  Quantitative  MOPs  will  be  used  whenever  possible.  Quantitative 
measures,  such  as  those  used  to  evaluate  the  timeliness  COI,  will  be  compared 
based  on  a  specified  numeric  measurement.  However,  due  to  the  fact  that  there 
exists  little  formal  criteria  for  the  decision  aids  to  be  evaluated  against,  qualitative 
MOPs  will  be  required  for  most  of  the  performance  measures.  Qualitative 
measures,  such  as  those  used  to  evaluate  the  tactical  accuracy  COI,  will  be 
evaluated  based  on  subjective  measures  determined  by  the  user.  There  will  be 
at  least  one  MOP  associated  with  each  MOE.  To  meet  the  needs  of  the 
operational  test  MOP’s,  multiple  data  requirements  have  been  derived  to  provide 
the  necessary  information  to  make  the  comparisons  and  resolve  each  COI. 

4.  Data  Requirements 

A  test  plan  identifies  what  data  requirements  are  needed  and  how  they  will 
be  collected.  The  data  requirements  are  necessary  to  compute  each  of  the 
MOPs.  In  the  case  of  this  operational  test  plan,  data  requirements  are  recorded 
using  data  collection  sheets  and  questionnaires.  These  documents  along  with  a 
complete  copy  of  the  test  plan  appear  in  Appendix  2  of  this  thesis.  A  dendritic 
analysis  can  show  the  traceable  relationship  between  COI’s,  MOE’s,  MOP’s,  and 
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data  requirements.  Figure  8  shows  the  dendritic  analysis  for  the  COI,  Mission 
Risk  Assessment  and  its  relatable  MOE’s,  MOP’s,  and  Data  Requirements. 


Figure  8  Dendritic  Method  for  COI,  Mission  Risk  Assessment. 


E.  DECISION  AID  MOE’S,  MOP’S,  AND  DATA  REQUIREMENTS 

In  this  section,  for  each  COI,  the  appropriate  MOE’s,  MOP’s,  and  data 
requirements  (DR)  necessary  to  complete  a  quantitative  or  qualitative  analysis  of 
the  decision  aids  as  a  whole  will  be  discussed. 

1.  Do  the  decision  aids  properly  conduct  target  prioritization  of 
targets  to  ensure  mission  accomplishment  of  higher  priority 
targets  before  lower  priority  targets? 

MOE  1.1  -  Accurate  target  selection.  The  MOE,  accurate  target  selection, 
is  accomplished,  if  during  post  decision  analysis  all  targets  of  higher  priority  are 
assigned  assets.  Specifically,  if  all  targets  cannot  be  assigned  assets,  than 
those  that  remain  unassigned  are  of  lower  priority. 
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MOP  1.1.1  -  Percent  properly  selected.  The  MOP  will  be  to  determine  the 
proportion  assigned  without  violating  the  established  priorities  of  the  targets. 

DR  1.1. 1.1  -  Total  missions  assigned. 

DR  1.1. 1.2  -  Number  of  missions  not  assigned  with  a  higher  priority  by  the 
decision  aids. 

DR  1.1. 1.3  -  Number  of  missions  not  assigned  by  decision  maker  (DM) 
without  the  aid  of  an  automated  tool. 

2.  Do  the  decision  aids  properiy  recognize  the  assets  that  are 
avaiiabie  for  a  particuiar  target? 

MOE  2.1  -  Asset  availability  accuracy.  The  MOE,  accuracy  of  asset 
consideration,  is  defined  by  whether  or  not  the  proposed  assignments 
consistently  reflect  those  that  are  available  for  assignment.  Knowledge  of  the 
assets  that  are  available  is  dependent  upon  the  accuracy  of  information  available 
within  the  command  and  control  centers.  For  this  reason  the  assumption  is 
made  that  the  information  within  the  TACC  is  both  current  and  accurate. 

MOP  2.1.1  -  Percent  available  considered.  The  MOP  will  be  to  determine 
the  percent  of  aircraft  missions  proposed  for  assignment  that  are  available. 

DR  2. 1.1.1  -  Number  of  aircraft  available  at  the  time  target  information  is 
received. 

DR  2. 1.1. 2  -  Number  of  aircraft  assigned  that  were  considered  available 
and  capable  by  the  DM. 

DR  2. 1.1. 3  -  Number  assigned  that  were  considered  available  and 
capable  by  the  DM  without  the  aid  of  an  automated  tool. 

3.  Do  the  decision  aids  properiy  recommend  targets  that  have 
the  abiiity  to  destroy  the  target? 

MOE  3.1  -  Target  Allocation  Accuracy.  The  MOE,  target  allocation 
accuracy,  is  defined  by  whether  or  not  the  assets  proposed  for  assignment  have 
the  ability  to  achieve  the  desired  affects  against  the  given  set  of  targets.  A 
correct  target  asset  pairing  for  this  MOE  reflects  the  proper  calculation  of 
munitions  effectiveness  against  the  given  target  types.  This  evaluation  was 
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conducted  on  the  unclassified  level  so  these  values  may  not  reflect  the  most 
accurate  capabilities  of  the  weapon  system. 

MOP  3.1.1  -  Percent  Correct  Assignments.  The  MOP  is  defined  as  the 
percentage  of  correct  assignments.  Assignments  will  be  considered  incorrect  if  a 
previously  established  probability  of  destruction  is  not  met  by  the  assignment. 
The  data  requirements  necessary  to  conduct  this  assessment  are: 

DR  3. 1.1.1-  Number  of  assignments  made  by  the  decision  aids  that  meet 
the  destruction  criteria. 

DR  3. 1.1.1-  Number  of  assignments  made  by  the  DM  that  meet  the 
destruction  criteria. 

DR  3. 1.1. 2-  Total  number  of  assignments  proposed. 

4.  Do  the  decision  aids  accurateiy  assess  the  risk  of  its  proposed 
assignments? 

MOE  4.1  -  Risk  assessment  accuracy.  The  MOE  is  defined  by  whether  or 
not  the  level  of  risk  depicted  by  the  decision  aids  is  accurate  for  each  platform 
when  a  surface-to-air  threat  is  associated  with  the  given  set  of  targets.  The 
accuracy  of  the  assessment  is  reliant  on  the  known  probabilities  of  kill  for  threat 
systems  against  various  aviation  platforms.  Again,  the  values  used  to  perform 
this  assessment  are  on  the  unclassified  level. 

MOP  4.1.1  -  Percent  properly  assessed.  The  percent  accurately 
assessed  is  qualitative  in  nature  and  a  success  will  be  determined  by  the 
MAWTS-1  Instructor  during  post  event  analysis.  The  data  required  to  perform 
the  assessment  of  the  first  MOP  is: 

DR  4.1. 1.1  -  Computed  risk  assessment  by  the  decision  aids. 

DR  4.1. 1. 2  -  Risk  assessment  by  DM. 

DR  4. 1.1. 3  -  Total  number  determined  to  be  accurate. 

MOE  4.2-  Accurate  SEAD  recommendation  based  on  risk  assessment. 
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MOP  4.2.1  -  Percent  of  proper  SEAD  recommendations.  The  second 
MOP,  SEAD  recommendation,  measures  whether  or  not  SEAD  is  proposed  in 
addition  to  the  asset  target  pairing  given  that  it  is  determined  necessary.  The 
data  required  to  conduct  this  assessment  is: 

DR  4.2.1-  Number  of  SEAD  assignments  made  when  required  by  decision 

aids. 

DR  4.2.1  -  Number  of  SEAD  assignments  made  when  required  by  the 
decision  maker. 

DR  4.2.2-  Total  number  of  SEAD  assignments  required. 

5.  Do  the  decision  aids  minimize  the  number  of  changes  to  the 
ATO  to  achieve  mission  accompiishment? 

MOE  5.1  -  ATO  Persistence.  The  MOE,  ATO  persistence,  is  defined  as 
the  level  of  impact  the  proposed  assignments  have  on  the  current  plan.  The 
fewer  the  number  of  physical  actions  required  by  the  decision  maker  the  more 
desirable  the  solution. 

MOP  5.1.1  -  Average  number  of  changes.  This  MOP  is  the  average 
number  of  changes  required  by  each  method.  For  each  decision  aid,  the 
changes  will  be  counted  and  compared  to  each  other. 

DR  5.1. 1.1  -  Number  of  changes  required  perTST  event. 

DR  5.1 .1 .2  -  Number  of  TST  events. 

6.  Do  the  decision  aids  provide  a  proposed  soiution  fast  enough 
to  be  effective  when  compared  to  current  methods? 

MOE  6.1  -  Decision  Speed.  The  MOE,  decision  speed,  is  the  time 
targeting  information  is  received  in  the  TACO  until  the  decision  aids  provides 
output  or  the  decision  maker  proposes  a  solution. 

MOP  6.1.1  -  Average  time  of  output  generation.  The  average  time  of 
target  info  input  and  decision  aid  output.  The  following  are  a  list  of  the  data 
requirements  needed  to  complete  the  calculations. 

DR  6. 1.1.1  -  Time  from  receipt  of  new  target  information  by  the  decision 
aids  or  decision  maker. 
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DR  6. 1.1. 2  -  Time  of  output  determined  by  decision  aids  or 
recommendation  on  a  particular  course  of  action  by  the  decision  maker. 

7.  Do  the  decision  aids  provide  the  decision  maker  with 
muitipie  aiternative  soiutions  for  mission  re-tasking  when 
avaiiabie? 

MOE  7.1  -  Multiple  alternative  generation.  The  single  MOE  for  this  issue, 
will  allow  us  to  compare  each  decision  aid  with  each  other  and  the  decision 
maker  to  determine  whether  or  not  each  method  provides  the  decision  maker 
with  an  acceptable  number  of  proposed  solutions. 

MOP  7.1.1  -  Average  number  of  alternatives  per  TST  event. 

DR  7.1. 1.1  -  Number  of  alternatives  recommended  by  the  decision  aids. 

DR  7.1. 1.2  -  Number  of  alternatives  provided  to  DM. 

DR  7.1 .1 .3-  Number  of  TST  events. 

8.  Does  the  decision  aid  output  provide  tacticaiiy  acceptabie 
assignments? 

MOE  8.1  -  Probability  of  correct  output  is  tactically  acceptable.  This  MOE 
is  qualitative  in  nature  and  the  correctness  of  the  proposed  solution  will  be 
determined  by  MAWTS-1  instructors. 

MOP  8.1.1  -  Percent  output  that  is  tactically  acceptable. 

DR  8.1 .1 .1  -  Number  of  acceptable  assignments. 

DR  8.1. 1.2-  Total  number  of  assignments  made. 

9.  Do  the  decision  aids  operate  correctiy  with  the  current  C4I 
architecture? 

MOE  9.1  -  Probability  of  properly  receive  the  ATO.  The  MOE  will 
determine  if  the  each  decision  aid  can  receive  the  basic  information  required  to 
perform  all  necessary  calculations.  The  inability  to  do  this  will  not  allow  the 
decision  aids  to  operate. 

MOP  9.1.1  -  Percent  of  ATOs  received  properly. 
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DR  9. 1.1.1  -  Number  of  Air  Tasking  Orders  properly  received  and 
interpreted  by  the  decision  aids. 

DR  9.1 .1 .2  -  Number  of  Air  Tasking  Orders. 

10.  Can  the  decision  aids  operate  continuously  without 
interruption  or  failure  for  a  prolonged  period  of  time? 

MOE  10.1  -  Reliability.  The  MOE  will  help  determine  if  the  decision  aids 
can  perform  during  sustained  operations  without  interruption.  A  final 
determination  for  this  COI  will  not  be  attained  during  this  test  alone.  The 
maximum  length  of  each  tactical  evolution  will  be  8  hours. 

MOP  10.1.1  -  Mean  time  between  failure.  The  MOP  will  be  calculated  by 
dividing  the  total  time  of  operation  by  the  number  of  internal  failures.  The  types 
of  failures  will  also  be  recorded. 

DR  10.1.1.1  -  Total  time. 

DR  10.1.1.2  -  Number  of  failures. 

DR  10.1.1.3  -  Types  of  failures. 

11.  Can  the  typical  user  enter  inputs  and  understand  the  outputs 
in  an  efficient,  intuitive  manner  with  minimal  training. 

MOE  11.1  -  Interface  usability.  The  MOE  will  be  evaluated  qualitatively 
with  questionnaires  and  interviews. 

MOP  11.1.1  -  Percent  operators  satisfied  with  the  interfaces. 

DR  11.1.1.1  -  Number  satisfied  with  the  probability  of  destruction  display. 

DR  11.1.1.2  -  Number  satisfied  with  the  risk  assessment  display. 

DR  11.1.1.3  -  Number  satisfied  with  the  persistence  measurement 
display. 

DR  11.1.1.4  -  Number  satisfied  with  the  current  mission  number  display. 

DR  11.1.1.5  -  Number  satisfied  with  the  ordnance  capability  display. 

DR  11.1.1.6  -  Number  satisfied  with  the  display  regarding  the  previously 
assigned  target,  its  priority,  and  associated  threat. 
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DR  11.1.1.7-  Number  satisfied  with  the  display  regarding  the  new  target 
and  its  priority  and  associated  threat 

DR  11.1.1.8  -  Total  number  of  operators 

F.  CRITERIA 

1.  Introduction 

The  operational  test  criteria  are  expressions  of  the  operational  level  of 
performance  of  the  decision  aids  required  by  the  typical  user  personnel  to 
demonstrate  mission  effectiveness  for  specific  functions  during  each  test.  The 
test  criteria  are  not  pass/fail  conditions,  rather,  they  represent  a  baseline  for  use 
in  the  design  and  evaluation  for  the  operational  test.  In  applying  the  test  criteria 
to  technical  initiatives,  it  should  be  kept  in  mind  that  ultimately  performance 
should  be  achieved  in  a  realistic  operational  environment, 

Without  the  aid  of,  a  specific  quantitative  requirement,  or  criteria  for 
comparative  analysis,  an  effective  evaluation  of  the  decision  aids  is  very  difficult. 
The  decision  aids  are  being  evaluated  without  any  formal  criteria  established. 
With  this  in  mind,  evaluating  the  overall  worth  of  the  decision  aids  are  limited  to: 
1)  comparison  to  manual  TST  methods  previously  discussed  in  chapter  two,  2)  a 
pairwise  comparison  of  the  decision  aids  developed  by  SPAWAR  and  NPS  with 
respect  to  the  COI’s,  MOE’s,  and  MOP’s. 

2.  Test  Criteria 

The  criterion  for  this  operational  test  is  first  the  comparative  analysis 
between  the  results  produced  by  the  decision  aids  and  the  manual  methods  of 
assigning  assets  to  TST’s  in  a  live  tactical  exercise.  The  agency  asserting  the 
manual  methods  during  this  process  is  the  Marine  Tactical  Air  Command  Center 
(TACC).  During  the  operational  test  the  TACC  will  be  manned  by  qualified 
Marine  personnel  experienced  in  the  Time  Sensitive  Targeting  process.  Each 
TST  event  will  provide  data  for  comparative  analysis  between  the  output  of  each 
of  the  decision  aids  and  the  manual  output. 
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The  second  criteria,  pairwise  comparison,  will  be  a  quantitative  and 
qualitative  analysis  of  the  two  decision  aids  developed  by  SPAWAR  and  NPS. 
The  comparison  will  be  conducted  over  each  TST  event  to  explore  differences 
and  advantages  of  each  decision  aid. 

G.  GENERAL  TEST  OPERATIONS  AND  OPERATIONAL  SCENARIO 

1.  Introduction 

The  development  of  test  scenarios,  segments,  and  trials,  is  the  next  step  in 
the  test  planning  process.  Although  overarching  scenarios  will  vary  from  Service 
to  Service  and  test  to  test,  it  is  the  segmenting  and  structuring  of  various 
segments  and  trials  (within  the  scenario)  to  identify  factor  effects  that  are  most 
important.20 

The  overall  test  should  be  concept  driven.  That  is,  it  needs  to  reflect  the 
test  objectives  that  answer  or  resolve  the  critical  operational  issues  for  the 
decision  aids.  As  it  pertains  to  this  evaluation,  the  test  needs  to  reflect  realistic 
TST  events  in  a  threat  environment.  Each  of  these  events  is  designed  to  foster 
appropriate  decision  making  in  response  the  TSTs  in  a  tactical  environment. 

The  tradeoff  is  between  controlling  scenarios  where  some  realism  is  lost, 
and  having  total  freeplay  occur  where  factor  confounding  will  significantly  limit 
evaluation  capability.2'i  The  exercise  control  group,  or  white  force,  when 
executing  the  overall  operational  scenario,  will  provide  the  method  of  control. 
The  white  force  can  systematically  vary  factors  and  conditions  within  TST  events 
without  the  TACC  knowing  and  introducing  bias.  In  developing  these  TST 
events,  some  factors  may  be  held  constant  and  others  varied  for  each  TST.  This 
will  allow  better  identification  and  analysis  of  factors  that  have  an  important 
influence  on  mission  accomplishment,  such  as  the  presence  of  surface-to-air 
weapons  systems. 


20  Fundamentals  of  Military  Experiment  Planning,  Design  and  Analysis,  by  Thomas  Hoivik, 
Naval  Postgraduate  School,  2002 


2'!  Fundamentals  of  Military  Experiment  Planning,  Design  and  Analysis,  by  Thomas  Hoivik, 
Naval  Postgraduate  School ,2002 
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2.  Evaluation  and  Scenario  Description 

The  evaluation  was  conducted  during  the  Weapons  Tactics  Instructor’s 
Course  (WTI)  2-04.  The  course  is  executed  by  Marine  Aviation  Weapons  and 
Tactics  Squadron  One  (MAWTS-1)  located  at  Marine  Corps  Air  Station  (MCAS) 
Yuma,  Arizona.  The  tools  and  methods  to  be  evaluated  will  be  located  in  the 
Marine  Tactical  Air  Command  Center  (TACC).  The  TACC  is  the  senior  aviation 
command  and  control  agency  for  the  Marine  Corps  and  is  ultimately  responsible 
for  the  conduct  of  the  air  war,  to  include  real  time  re-tasking  of  aviation  assets. 
WTI  is  designed  to  train  and  evaluate  the  Marine  Corps  experienced  Aviators, 
Command  and  Control  Officers,  and  Aviation  Ground  Support  Officers  in  the 
conduct  of  their  assigned  specialties.  Once  training  is  complete  the  officers  and 
enlisted  personnel  are  sent  back  to  their  units  to  become  the  resident  experts  for 
the  unit.  Within  the  Marine  Corps  aviation  community,  WTI  has  been  referred  to 
as  a  “graduate  level  course  for  tactics.” 

The  execution  of  a  WTI  course  presents  a  unique  opportunity  to  evaluate 
not  just  our  Marines,  but  the  tools  developed  to  support  the  conduct  of  their 
mission.  The  course  is  divided  into  two  phases:  Academic  and  Flight.  The  flight 
phase  of  the  course  provides  the  vehicle  for  this  evaluation. 

During  the  flight  phase  the  evaluation  focuses  on  only  two  separate  flight 
phase  evolutions.  The  MACCS  Integrated  Simulated  Training  Exercise 
(MISTEX),  and  the  Final  Exercise  (FINEX)  will  comprise  the  specific  test  vehicles 
for  the  evaluation. 

MISTEX  is  a  static  exercise  consisting  of  the  entire  Marine  Aviation 
Command  and  Control  Decision  aids  in  the  field  executing  a  realistic,  low 
intensity,  tactical  scenario  over  a  two-hour  period.  This  MISTEX  scenario  is 
executed  twice  over  a  two-day  period  to  ensure  the  MACCS  is  fully  operational 
prior  to  the  beginning  of  flight  operations. 

FINEX  is  the  culminating  exercise  of  the  WTI  course.  It  is  highly  intensive 
and  demonstrates  all  six  functions  of  Marine  Aviation.  These  six  functions 
include  Control  of  Aircraft  and  Missiles,  Anti-Air  Warfare,  Offensive  Air  Support, 
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Assault  Support,  Air  Reconnaissance,  and  Electronic  Warfare.  The  functions  are 
driven  by  a  tactical  scenario  where  command  and  control  officers  are  exercising 
their  delegated  authority  in  the  direction  and  control  of  aircraft.  The 
determination  of  whether  or  not  the  decision  aids  can  adequately  support  the 
decision  makers  in  this  type  of  operation  was  assessed  during  this  evolution. 

The  scenario,  for  the  purposes  of  the  evaluation,  consists  of  offensive  and 
defensive  combat  operations  involving  ground  and  air  forces  against  a  heavy 
mechanized  division  in  a  desert  environment.  The  enemy  air  capabilities  consist 
of  two  airforce  groups  with  a  mixture  reconnaissance,  anti-air,  air  to  ground,  and 
support  capabilities.  In  addition,  a  major  consideration  of  the  friendly  forces  was 
the  enemy’s  surface-to-surface,  and  surface  to  air  missile  threat.  The  phase  of 
the  operation  in  which  the  decision  aids  are  evaluated  primarily  involved 
offensive  air  support  and  assault  support  operations.  The  general  description  of 
the  enemy’s  capabilities  provide  the  source  of  targets  and  threats  for  evaluation 
of  the  decision  aids.  The  enemy  forces  are  simulated  on  the  ground  either 
through  the  use  of  static  targets  or  electronic  threat  emitters.  Friendly  aviation 
support,  command  and  control,  and  air  defense  forces  used  in  the  execution  of 
this  scenario  was  live.  Friendly  ground  combat  forces  will  be  represented  by, 
one  USMC  infantry  company,  and  one  USMC  Artillery  battery.  There  will  be  no 
notional/simulated  friendly  aircraft  during  the  execution  of  the  flight  phase.  The 
MAWTS-1  Test  Plan  document  is  include  in  Appendix  2  of  this  thesis. 

3.  Data  Recording 

The  results,  for  each  event,  will  be  collected  by  test  personnel.  Test 
personnel  are  separate  and  distinct  from  the  operational  force.  The  data  was 
compiled  using  questionnaires  and  data  collection  sheets  as  shown  in  the 
MAWTS-1  Test  Plan,  Appendix  2.  The  results  and  analysis  of  the  operational 
test  are  presented  in  Chapter  V. 

4.  Operating  Forces 

The  operational  force  directly  relating  to  the  evaluation  is  representative  of 
Marine  Air  Command  and  Control  System,  approximately  900  Marines.  The 
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representative  air  wing,  which  supports  all  aviation  operations  within  the 
scenario,  provides  approximately  100  daily  aviation  sorties. 
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V.  RESULTS  AND  DISCUSSION 


A.  INTRODUCTION 

On  April  16,  2004,  the  test  and  evaluation  of  the  decision  aid  concluded 
with  the  completion  of  the  FINEX  3  evolution  of  the  Weapons  Tactics  Instructors 
Course  (WTI).  During  the  five  flight  phase  evolutions;  MISTEX  1,  MISTEX  2, 
FINEX  1,  FINEX  2,  and  FINEX  3,  twenty  three  Time  Sensitive  Targets  emerged 
as  part  of  the  tactical  scenario.  Detailed  data  with  respect  to  the  COI’s  discussed 
in  Chapter  IV  were  collected  with  the  exception  of  COI  11,  Fluman  Factors,  due 
to  the  high  tempo  of  exercise  operations. 

Originally,  it  was  the  intent  of  the  evaluation  to  compare  the  results  of  the 
SPA  WAR’S  genetic  algorithm.  Unfortunately,  due  to  software  difficulties  their 
decision  aid  was  unavailable  to  be  evaluated  adequately. 

The  following  sections  of  this  chapter  provide  the  answers  to  each 
question  posed  by  the  COI’s  provided  in  the  test  plan. 

B.  COI  1:  TARGET  SELECTION.  DO  THE  DECISION  AIDS  PROPERLY 

CONDUCT  TARGET  PRIORITIZATION  OF  TARGETS  TO  ENSURE 

MISSION  ACCOMPLISHMENT  OF  HIGHER  PRIORITY  TARGETS 

BEFORE  LOWER  PRIORITY  TARGETS? 

This  COI  addresses  the  ability  of  the  decision  aid  to  properly  conduct 
target  prioritization  by  assigning  assets  to  only  the  highest  priority  targets. 
During  each  evolution  a  prioritized  target  list  was  provided.  During  the  MISTEX 
evolutions  the  predominant  number  of  ATO  assignments  were  a  pre-planned 
mission  against  prioritized  targets.  However,  during  the  FINEX  evolutions  the 
majority  of  air  missions  were  in  general  support  and  on  alert  status.  A  violation 
or  miss-assignment  as  identified  in  Table  1  occurs  if: 

1 .  A  TST  was  ignored,  and  not  assigned  an  asset. 

2.  Of  the  set  targets  addressed  during  the  time  horizon  defined  by  the 
TST’s  vulnerability  window,  a  higher  priority  target  was  left 
unassigned. 

Table  2,  provides  all  data  obtained  with  respect  to  this  COI. 
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Table  2  COI  1,  Target  Selection.  Depicts  the  data  required  to 
determine  percentage  of  target  selection  done  accurately. 


Based  on  the  data  collected  the  NPS  Decision  Aid  provided  accurate 
target  selection  100%(34/34)  of  the  time.  By  comparison  the  decision  maker 
accurately  selected  the  targets  to  be  assigned  assets  76  %  (13/17)  of  the  time. 
The  difference  in  TST  events  between  the  decision  aid  and  the  decision  maker  is 
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the  result  of  one  of  the  following  reasons:  target  location  was  not  accurate,  or  the 
target  was  mistakenly  not  assigned  an  asset.  Of  the  miss-assignments  that 
occurred  the  following  are  cited  as  examples.  In  the  first  TST,  the  decision 
maker  chose  to  ignore  the  target  and  execute  the  ATO  as  planned.  In  the 
seventh  TST,  occurring  during  MISTEX  2,  the  decision  maker  diverted  a  mission 
assigned  to  a  target  of  priority  3,  while  an  asset  on  ground  alert  remained 
available  with  the  capability  to  destroy  the  target.  In  some  cases  the  decision 
maker  made  no  assignment.  This  action  did  not  count  for  or  against  the  decision 
maker,  because  it  indicates  an  option  selected  other  than  an  aviation  asset. 

Within  the  scope  of  the  evaluation  it  was  determined  that  when  given  a 
prioritized  list  of  targets  the  NPS  Decision  Aid  does  accurately  select  targets  for 
assignment  as  required  by  COI  1.  It  is  recommended  that  no  changes  be  made 
to  the  model  with  respect  to  target  selection. 

C.  COI  2:  ASSET  AVAILABILITY.  DO  THE  DECISION  AIDS  PROPERLY 

RECOGNIZE  THE  ASSETS  THAT  ARE  AVAILABLE  FOR  A 

PARTICULAR  TARGET? 

The  COI  addressed  the  decision  aids  ability  to  select  assets  available  for 
assignment.  An  asset  not  available  falls  into  to  one  of  the  following  categories: 

1 .  Asset  canceled  on  the  ATO  due  to  mechanical  problems. 

2.  Asset  that  has  proceeded  to  target. 

3.  Asset  that  is  not  under  control  of  the  component  reassigning  it. 

Availability  is  based  on  the  accuracy  of  the  information  within  the  TACC.  If 
the  information  available  to  the  decision  maker  and  the  decision  aid  is  inaccurate, 
neither  are  be  held  accountable  for  a  proposal  based  on  the  available 
information.  Table  3,  depicts  the  data  obtained  from  the  proposals  made  by  the 
decision  aid  and  the  decision  maker. 
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Finex3  Finex2  Finex  1  MISTEX2  MISTEX 1 


COI  #2  Asset  Availability 

TST  #  of  Selceted  for  assignment  that  were  unavailable 


_ NFS _ _ DM _ 

#Unavail  Total  Assign  #UnAvail  Total  Assign 


Table  3.  COI  2,  Asset  Availability,  depicts  the  number  of  assets  that 
were  proposed  for  assignment  by  the  test  subjects  that  were  unavailable. 


Based  on  the  manual  updates  allowed  by  the  interface  the  NPS  decision 
was  able  to  discern  which  assets  were  available  for  assignment.  Assuming  the 
information  within  the  TACC  was  accurate  the  NPS  decision  aid  proposed  assets 
that  were  available  for  assignment  100%  (34/34)  of  the  time,  the  decision  maker 
employing  manual  methods  did  so  94%  (16/17)  of  the  time. 
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Based  on  the  scope  of  this  evaluation  the  decision  aid  successfully 
answered  COI  2,  Asset  Availability. 

D.  COI  3.  TARGET  ASSET  PAIRING.  DO  THE  DECISION  AIDS 
PROPERLY  RECOGNIZE  THE  ASSETS  THAT  ARE  AVAILABLE  FOR 
A  PARTICULAR  TARGET? 

This  COI  addressed  the  ability  of  the  assets  proposed  for  assignment  by 
the  decision  aid  to  destroy  the  target.  In  each  event  a  determination  was  made 
as  to  whether  or  not  the  required  probability  of  destruction  for  each  target  was 
achieved.  The  desired  probability  of  destruction  was  provided  by  the  MAWTS-1 
Instructors.  Any  asset  that  was  assigned  that  could  not  achieve  the  destruction 
criteria  was  considered  a  bad  assignment.  The  scoring  of  this  issue  was  done 
qualitatively  by  the  MAWTS-1  Instructors.  The  results  are  depicted  in  Table  4 
below. 
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COI  #3  Target  Asset  Planning 

TST 

Number  of  Selected  for  assignment  to  primary  target  that  meet  destruction  criteria 

NPS 

DM 

#  properly 
selected 

Total  Assign 

#  properly 
selected 

Total  Assign 
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LU 
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1 
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2 

1 

1 

1 

1 

3 

3 

3 

0 

0 

4 

1 

1 

1 

1 

5 

1 

1 

1 

1 

Table  4  COI  3,  Results. 


In  resolving  this  COI,  a  determination  of  whether  or  not  the  decision  aid 
met  the  established  criteria  was  simplified  because  the  probability  of  successfully 
destroying  the  target  is  provided  as  part  of  the  output  seen  in  Figure  9,  below. 
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This  figure  provides  an  example  of  the  results  achieved  during  the  evaluation  of 
TST  5,  during  MISTEX  1. 


- w - 

1  ^ 

1  1 

- - 

- *= - 

1  ^  ~ 

1  '■  ~ 

1  '  1 

1  ^  1 

Missions 

1 

Recommend  | 

Clear  | 

Indude 

Mission 

C/S 

ETD 

ETO 

# 

Aircraft 

Configuration 

Assignment 

Recommend 

6111 

Razor  11 

15:51 

17:07 

2 

AV8B 

[21  CBU58 

TGT  SA3 

6141 

Razor  41 

15:51 

17:07 

2 

AV8B 

[21  CBU58 

TGT2 

6143 

Razor  43 

15:51 

17:07 

2 

AV8B 

[41  MK83 

TGT  2 

6115 

Razor  15 

15:51 

17:07 

2 

AV8B 

[41  MK83 

TGT  3 

X 

6145 

Razor  45 

15:51 

17:07 

2 

AV8B 

[41  MK83 

SLA  15 

SA8 

6147 

Razor  47 

15:51 

17:07 

2 

AV8B 

[6]  MK83 

SLA  15 

X 

6161 

Razor  61 

15:51 

17:07 

2 

AV8B 

[41  MK83 

KB  AA80 

X 

6101 

Storm  01 

16:19 

19:09 

2 

EA6B 

[1]  POD  [11  HARM 

TGT  SA3 

) 

X 

6103 

Storm  03 

16:19 

19:09 

1 

EA6B 

[1]  POD  [1]  HARM 

TGT  2 

X 

6104 

Storm  04 

16:19 

19:09 

1 

EA6B 

[1]  POD  [11  HARM 

SLA  30 

) 

X 

6131 

Latch  31 

16:34 

17:54 

2 

F18D 

[4]  MK83 

TGTl 

TGTl 

5 

X 

6133 

Latch  33 

16:34 

17:54 

2 

F18D 

[41  MK83 

TGTl 

1 

6135 

Latch  35 

16:34 

17:54 

2 

F18D 

[4]  MK83 

TGT  3 

) 

6137 

Latch  37 

17:04 

18:24 

2 

F18D 

[4]  MK83 

KB  AA80 

5 

6175 

Latch  75 

17:49 

19:09 

2 

F18D 

[4]  MK83 

KB  AA80 

7 

6163 

Razor  63 

18:07 

19:21 

2 

AV8B 

[4]  MK82 

SLA  15 

) 

) 

) 

1 

6165 

Razor  65 

18:22 

19:36 

2 

AV8B 

[2]  CBU58 

SLA  15 

Figure  9  TST  5,  MISTEX  1  result  shows  the  recommended 
assignment,  in  the  far  right  corner,  of  a  section  of  AV8B  Harriers  to  the 
target  (SA-8). 


The  proposed  assignment  by  the  decision  aid  of  an  AV-8B  to  strike  the 
SA-8  represents  a  solution  that  achieves  a  probability  of  success  of  94%.  This  is 
depicted  in  Figure  10,  a  screen  shot  of  the  interface  for  the  decision  aid. 
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Targets 


ProbabCty  ProbabSty  ProbabCty 
Indude  Target  Priority  Type  Threat  of  Success  Achieved  of  Attrition 


Figure  10.  Target  Pairing  Results.  The  decision  aid  determined  that 
by  assigning  a  AV-8B  Harrier  to  SA-8  a  probability  of  success  of  94%  would 
be  achieved.  The  required  probability  of  success  was  only  75%. 


This  same  target  addressed  by  the  decision  maker  with  no  automated 
support  was  resolved  by  assigning  an  EA-6B  Prowler  and  a  FA-18D  with  an 
ordnance  configuration  consisting  of  LAU-61.  In  this  case,  this  represents  an 
inadequate  solution  as  determined  by  the  MAWTS-1  instructors.  The  reason  for 
the  determination,  LAU-61  s  are  rockets  primarily  used  for  marking  targets,  and 
are  too  inaccurate  for  a  high  probability  of  success  to  be  achieved. 

The  test  results  indicated  that  the  decision  aid  proposed  a  solution  that 
met  the  required  probability  of  destruction  100%  (34/34)  of  the  time.  It  should  be 
noted  that  the  decision  aid  was  operated  on  an  unclassified  system  so  known 
probabilities  of  kill  used  in  these  calculations  were  also  unclassified.  Additionally, 
the  decision  aid  did  not  take  into  account  the  implications  of  weather  on 
ordnance  selection,  but  this  was  not  a  factor  in  the  evaluation. 

The  decision  maker’s  proposed  assignment  was  also  evaluated 
qualitatively.  The  MAWTS-1  instructors  determined  that  the  accuracy  of  each 
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assignment,  and  as  a  result,  the  decision  maker  achieved  the  desired  probability 
of  success  82%  (14/17)  of  the  time. 

Based  on  the  scope  of  this  evaluation  the  NPS  Decision  Aid  met  the  COI 
requirements.  It  is  recommended  that  in  further  testing  classified  values  for 
probability  of  kill  be  used  while  employing  the  decision  aid. 

E.  COI  4.  RISK  ASSESSMENT.  DO  THE  DECISION  AIDS  ACCURATELY 

ASSESS  THE  RISK  OF  PROPOSED  ASSIGNMENTS? 

This  addresses  whether  or  not  the  decision  aid  properly  computed  the 
level  of  risk  to  aircraft  given  the  presence  of  Surface-to-Air  Threats.  Additionally, 
based  on  a  determination  of  risk,  an  assessment  of  whether  or  not  SEAD  was 
appropriately  recommended  was  also  conducted. 

Table  5,  provided  below,  presents  the  results  pertaining  to  this  COI. 
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COI  #4  Risk  Assessment 
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Table  5.  Risk  Assessment.  The  table  identifies  for  each  assignment 
the  probability  or  risk  of  losing  an  aircraft  as  a  result  of  a  Surface-To-Air 
Threat. 


In  each  case  the  same  assessment  was  similar  for  both  the  decision  aid 


and  the  unaided  decision  maker.  The  probabilities  of  an  aircraft  being  destroyed 
(attrition)  are  based  on  unclassified  data.  In  Table  6  below,  a  determination  for 
each  target  for  the  requirement  of  SEAD  is  made.  Using  this  determination  a 
comparison  is  conducted  to  determine  the  rate  at  which  SEAD  was  properly 
recommended.  The  data  reveals  that  the  decision  aid  recommended  SEAD 
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correctly  80%  (4/5)  of  the  time.  By  comparison  the  decision  maker  also 
recommended  BEAD  80%  (4/5)  of  time. 


SEAD  Requirement 
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Table  6,  SEAD  Recommendation.  The  table  indicates  whether  or  not 
the  test  subjects  recommended  the  assignment  of  SEAD  when  it  was 
determined  necessary  by  the  MAWTS-1  Instructors. 


Of  note  is  TST  23,  in  this  case  the  decision  maker  assessed  the  risk  of 
losing  an  aircraft  to  be  60%,  based  on  the  capabilities  of  an  SA-6.  Although 
SEAD  was  recommended  in  this  case,  none  was  available.  The  decision  maker 
chose  to  assign  an  aircraft  and  accept  the  risk.  By  comparison,  the  decision  aid 
did  not  recommend  SEAD.  Instead  the  proposed  solution  was  to  assign  an 
available  section  of  AH-1W  helicopters.  Due  to  their  ability  to  use  terrain  to  mask 
themselves  from  the  threat,  the  risk  of  attrition  was  limited.  What  is  not  assumed 
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by  the  decision  aid  is  that  the  threat,  an  SA-6,  is  customarily  accompanied  by  a 
low  altitude  anti-aircraft  artillery  (AAA)  threat,  resulting  in  a  poor  assignment. 

Based  on  the  scope  of  this  evaluation  it  can  be  concluded  that  the 

decision  aid  accurately  assessed  risk  and  recommended  BEAD  when  required. 

It  is  recommended  that  the  decision  aid  should  be  modified  to  incorporate  a 

realistic  employment  of  enemy  threat  systems  when  employing  helicopter  assets. 

F.  COI  5.  PERSISTENCE.  DO  THE  DECISION  AIDS  MINIMIZE  THE 
NUMBER  OF  CHANGES  TO  THE  ATO  TO  ACHIEVE  MISSION 
ACCOMPLISHMENT? 

The  test  determined  if  the  decision  aid  minimized  the  number  of  changes 
for  each  TST  event  required  to  obtain  the  desired  effects  on  the  targets  being 
attacked  during  the  vulnerability  window.  The  impact  of  the  decision  on  the 
effectiveness  of  other  missions  was  not  evaluated,  therefore  averages  could  be 
misleading.  To  make  this  determination  the  decision  maker  would  have  to 
employ  each  recommendation  made  by  the  decision  aid.  This  did  not  fall  within 
the  scope  of  the  evaluation.  The  following  results,  displayed  in  Table  7,  provide 
a  basis  for  determining  the  average  number  of  changes  for  each  TST  event. 


64 


Table  7.  COI  5,  Persistence.  The  table  provides  the  number  of 
required  changes  to  be  made  by  the  decision  maker  for  each  TST  event  in 
order  to  execute  the  proposed  solution. 

From  the  table  we  can  conclude  that  on  average  the  decision  aid  solution 
required  1.7  (31/23)  changes  with  a  maximum  of  three.  The  decision  maker 
solution  on  average  required  0.75  changes  (17/23)  with  a  maximum  of  two.  In 
the  event  that  the  decision  maker  did  not  require  a  change  as  indicated  by  an  (*) 
in  Table  6,  one  of  two  possibilities  occurred;  the  decision  maker  chose  to  not 
strike  the  TST,  or  elected  to  address  the  target  using  indirect  fires.  Indirect  fires 
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are  not  an  option  for  the  decision  aid.  Furthermore  on  average  for  every  TST 
there  were  4.7  (110/23)  missions  available  for  assignment.  The  number  of 
available  missions  ranged  from  one  to  eight. 

Based  on  these  results  the  conclusion  is  that  the  decision  aid  does  appear 
to  minimize  the  number  of  changes  per  event,  but  requires  further  testing  to  be 
conclusive. 

G.  COI  6.  DECISION  SPEED.  DO  THE  DECISION  AIDS  PROVIDE  A 
PROPOSED  SOLUTION  FAST  ENOUGH  TO  BE  EFFECTIVE  WHEN 
COMPARED  TO  CURRENT  METHODS? 

This  COI  addresses  the  ability  of  the  decision  aid  to  propose  a  solution  in 
enough  time  to  strike  a  target  within  a  specified  vulnerability  window.  During  this 
evaluation,  no  vulnerability  window  was  specified  for  each  target.  A  vulnerability 
window  can  vary  depending  on  the  target.  The  default  vulnerability  window  for 
this  exercise  was  determined  to  be  20  minutes.  This  window  was  chosen 
because  it  provided  a  more  stringent  case  to  conduct  the  evaluation.  As  a  result 
the  decision  aid  was  held  to  a  tighter  requirement  than  the  decision  maker. 
Table  8  provides  the  decision  speed  for  each  event.  The  scoring  of  time  begins 
with  the  receipt  of  a  target  report,  it  concludes  with  the  proposal  of  a  decision  by 
the  decision  aid  and  the  decision  maker. 
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COI  #6  Timeliness 
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Table  8.  COI  6.  Decision  Speed.  The  table  provides  the  time  in 
seconds  for  the  test  subjects  to  recommend  a  solution  to  the  target. 


Based  on  the  results  depicted  in  the  table,  the  decision  aid  on  average 
provided  a  solution  in  1.7  minutes  (1167sec/23),  with  a  range  of  one  to  two 
minutes.  By  comparison  the  decision  maker  provided  a  solution,  on  average,  in 
8.4  minutes  (11160  sec/22),  with  a  range  of  one  minute  to  38  minutes.  By 
observation  there  appears  to  be  a  significant  military  difference. 

The  conclusion  is  that  the  decision  aid  on  average  does  provide  a 
recommendation  within  the  time  required  (2-3  minutes)  to  make  an  effective 
decision.  Therefore,  the  COI  has  been  satisfactorily  met. 
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H.  COI  7.  OPTIONS.  DO  THE  DECISION  AIDS  PROVIDE  THE  DECISION 

MAKER  WITH  MULTIPLE  ALTERNATIVE  SOLUTIONS  FOR  MISSION 

RE-TASKING  WHEN  AVAILABLE? 

Every  decision  maker  will  weight  the  effect  of  a  proposed  solution 
differently.  This  can  be  attributed  to  the  varying  background  and  experience  of 
each  decision  maker.  As  a  result  the  number  of  options  available  may  improve 
the  quality  of  the  decision.  This  COI  addresses  the  ability  of  the  decision  aid  to 
provide  the  decision  maker  with  different  courses  of  action. 

The  test  results  showed  the  decision  aid  was  only  able  to  provide  the 
decision  maker  with  one  course  of  action.  As  a  result,  for  each  TST  event,  the 
COI  was  not  satisfactorily  met.  This  can  be  attributed  to  the  design  of  the 
decision  aid.  It  is  recommended  that  the  decision  aid  be  modified  to  provide 
multiple  solutions. 

I.  COI  8.  TACTICALLY  ACCURATE.  DOES  THE  DECISION  AIDS’ 

OUTPUT  PROVIDE  TACTICALLY  ACCEPTABLE  ASSIGNMENTS? 

The  focus  of  this  COI  was  to  evaluate  if  the  results  provided  by  the 
decision  aid  represent  assignments  that  were  tactically  accurate.  The 
assessment  of  this  COI  was  qualitative  in  nature.  At  the  conclusion  of  each 
evolution,  the  MAWTS-1  instructor  reviewed  the  proposed  assignment  for  each 
TST,  and  a  decision  was  made.  In  each  of  the  twenty-three  events  the  instructor 
concluded  that  assignments  proposed  by  the  decision  aid  represented  tactically 
acceptable  solutions.  The  following  example  shows  an  instance  where  the 
decision  aid  solution  was  considered  accurate,  while  by  comparison,  the  decision 
maker’s  solution  was  not. 

During  FINEX  3  at  2104,  a  FROG-7  surface-to  surface  missile  system  was 
located  and  the  TACC  was  informed.  Figure  11,  depicts  the  assets  available  at 
the  time  the  target  emerged. 
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Figure  11.  FINEX  3  ATO.  The  assets  outlined  indicate  the  available 
assets  at  the  time  of  the  emerging  TST.  From  these  available  assets  a 
solution  can  be  proposed  to  strike  the  target. 


In  the  vicinity  of  the  target,  an  SA-8  was  known  to  exist.  Based  on  these 
inputs  the  decision  aid  determined  the  best  asset  for  assignment  to  be  mission 
6163,  a  section  of  AV-8B  Harriers,  that  had  launched  at  2145.  The  mission  was 
previously  assigned  in  general  support  of  CAS  missions  and  was  available  for 
assignment  to  a  higher  priority  mission.  By  comparison  the  decision  maker 
chose  to  re-assign  nothing,  citing  the  threat  as  the  factor.  During  post-evolution 
analysis  the  MAWTS-1  instructor  determined  that  the  SA-8  did  not  pose  a  viable 
threat  to  the  fixed  wing  aircraft  operating  at  altitude.  For  this  example  it  was 
concluded  that  the  decision  aid  proposed  the  correct  course  of  action,  while  the 
decision  maker  did  not. 

Based  on  the  scope  of  the  evaluation  it  is  determined  that  the  decision  aid 
meets  the  tactical  accuracy  requirements  of  this  COI  satisfactorily. 

J.  COI  9.  INTEROPERABILITY.  DO  THE  DECISION  AIDS  OPERATE 

CORRECTLY  WITH  THE  CURRENT  C4I  ARCHITECTURE? 

This  COI  addresses  the  ability  of  the  decision  aid  to  interoperate 
satisfactorily  in  the  tactical  environment  with  existing  systems.  During  the  course 
of  the  evaluation,  five  ATO  were  developed  and  transmitted  to  the  exercise  force. 
In  all  cases  the  decision  aid  was  able  to  accept  the  ATO  and  perform  its 
necessary  tasks.  The  ATO  parser  developed  to  perform  this  function  was 
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reviewed  by  the  unit  and  has  since  been  approved  for  incorporation  into 
everyday  operations.  Based  on  these  results  the  requirements  of  the  COI  have 
been  met.  However,  it  is  recommended  that  for  future  testing  that  the  decision 
aid  be  adapted  to  accept  real  time  updates  over  the  tactical  data  network.  This 
would  enhance  the  effectiveness  of  the  decision  aid  to  determine  available 
resources  at  a  given  moment.  This  process  is  currently  conducted  manually  and 
subject  to  error. 

K.  COI  10.  SOFTWARE  RELIABILITY.  CAN  THE  DECISION  AIDS 
OPERATE  CONTINUOUSLY  WITHOUT  INTERRUPTION  OR  FAILURE 
FOR  A  PROLONGED  PERIOD  OF  TIME? 

This  COI  addressed  the  decision  aids  ability  to  function  continuously 
without  interruption  over  the  course  of  each  evolution.  The  length  of  each 
evolution  ranged  from  two  to  seven  hours.  During  each  evolution  the  decision 
aid  operated  continuously  without  interruption.  It  can  be  concluded,  based  on  the 
limited  scope  of  this  evaluation  that  the  decision  aid  appears  to  meet  this  COI. 
For  a  final  determination  it  is  recommended  that  the  decision  aid  be  employed 
during  exercises  of  much  longer  duration. 

L.  CONCLUSION 

During  the  evaluation  ten  of  the  eleven  planned  COI’s  were  evaluated. 
The  overall  results  are  depicted  in  Table  9. 
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COI 

NFS  DECISION  AID  RESULT 

Adequate 

Inadequate 

Unresolved 

COI  1 

X 

COI  2 

X 

COI  3 

X 

COM 

X 

COI  5 

X 

COI  6 

X 

COI  7 

X 

COI  8 

X 

COI  9 

X 

COI  10 

X 

COI  11 

X 

Table  9  COI  Evaluation  Results. 


Within  the  scope  of  MISTEX/FINEX  evaluation  all  critical  issues 
addressed  were  satisfactorily  met  except  for  COI  7,  the  number  of  options 
provided  by  the  decision  aid.  COI  11,  Human  Factors,  was  not  able  to  be 
adequately  addressed  by  this  evaluation. 

M.  TEST  LIMITATIONS 

The  evaluation  was  limited  by  the  following  factors.  The  scenario  used  in 
the  evaluation  did  not  allow  for  targets  in  an  urban  environment.  The 
environment  did  not  incorporate  adverse  weather  effects,  which  limited  the 
evaluation  of  asset-target  pairing.  Specifically,  low  altitude  threat  systems  could 
be  countered  with  high  altitude  tactics,  therefore,  eliminating  the  requirement  for 
SEAD  in  most  cases.  Finally,  the  duration  of  the  exercise  limited  the  assessment 
of  software  reliability.  As  a  result  it  is  recommended  that  further  testing  include 
operations  of  at  least  24  hours  in  duration. 
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Chapter  VI,  provides  a  summary  of  the  conclusions  and  recommendations 
generated  by  the  MISTEX/FINEX  evaluations. 
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VI.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  CONCLUSIONS  AND  IMPACT 

The  emergence  of  Time  Sensitive  Targets  (TST)  poses  an  imminent  threat 
to  friendly  forces  and  the  successful  completion  of  the  friendly  commanders 
mission.  Failure  to  decide  and  act  within  a  short  period  of  time,  specified  by  a 
target’s  vulnerability  window,  results  in  loss  of  life  or  the  disruption  of  a  vital  area. 

The  tactical  decision  aid  developed  in  this  thesis  provides  the  decision 
maker  with  a  fast,  accurate,  and  appropriate  aviation  response  to  the  emergence 
of  a  TST.  The  decision  aid  employs  optimization  techniques  that  incorporate  all 
available  resource  information  provided  by  the  Air  Tasking  Order  (ATO), 
including:  the  number  and  type  of  aircraft,  ordnance  load,  time  window,  and  the 
aircrafts’  previously  assigned  task.  Coupled  with  commander’s  guidance  in  the 
form  of  a  prioritized  list  of  targets,  and  assessment  of  the  threat,  the  decision  aid 
determined  which  asset  to  assign  to  each  target  within  the  vulnerability  window. 

The  tactical  decision  aid  was  successfully  demonstrated  in  the  live  tactical 
environment  at  Marine  Aviation  Weapons  and  Tactics  Squadron  One,  Yuma, 
Arizona.  The  results  of  the  Critical  Operational  Issues  (COI)  addressed  by  the 
tactical  decision  aid  are  summarized  below: 

COM.  Target  Selection.  The  decision  aids  properly  conduct  target 

prioritization  of  targets  to  ensure  mission  accomplishment  of  higher 
priority  targets  before  lower  priority  targets. 

COI  2.  Asset  Availability.  The  decision  aids  properly  recognize  the 
assets  that  are  available  for  a  particular  target. 

COI  3.  Target  Asset  Pairing.  The  decision  aids  properly  recommend 
assets  that  have  the  ability  to  destroy  the  target. 

COI  4.  Mission  Risk  Assessment.  The  decision  aids  accurately  assess 
the  risk  of  its  proposed  assignments. 

COI  5.  Persistence.  The  decision  aids  minimize  the  number  of  changes 
to  the  ATO  to  achieve  mission  accomplishment. 

COI  6.  Timeliness.  The  decision  aids  provide  a  proposed  solution  fast 
enough  to  be  effective  when  compared  to  current  methods. 

COI  7.  Options.  The  decision  aids  did  not  provide  the  decision  maker  with 
multiple  alternative  solutions  for  mission  re-tasking  when  available. 

COI  8.  Tactical  Accuracy.  The  decision  aids  output  provide  tactically 

73 


acceptable  assignments. 

COI  9.  Interoperability.  The  decision  aids  operate  correctly  with  the 
current  C4I  architecture. 

CO1 10.  Software  Reliability.  The  decision  aids  operate  continuously 
without  interruption  or  failure  for  a  prolonged  period  of  time. 

The  conclusion  is  that  the  decision  aid  can  significantly  improve  combat 
effectiveness  against  the  emergence  of  a  TST. 

B.  RECOMMENDATIONS 

Based  on  the  results  of  the  test  and  evaluation  the  following 
recommendations  are  provided  to  enhance  future  performance  of  the  decision 
aid  in  a  tactical  environment. 

1.  Classified  Values 

The  decision  aid  during  the  test  and  evaluation  functioned  on  an 
unclassified  level.  Therefore,  it  is  recommended  that  future  employment  of  the 
decision  aid  employ  the  following  classified  values: 

a.  Ordnance  single  point  probability  of  kill. 

b.  Aircraft  Survival  Probabilities. 

c.  EA-6B  Prowler  Suppression  Probabilities. 

The  incorporation  of  more  accurate,  classified  values,  increases  the 
accuracy  of  the  overall  selection  of  the  asset  selected  to  strike  a  particular  target. 

2.  Rotary  Wing  Assets 

The  decision  aid  was  expanded  during  the  test  and  evaluation  to 
incorporate  the  inclusion  of  rotary  wing  aircraft  into  strikes  packages.  When 
considering  the  employment  of  rotary  wing  aircraft,  an  integrated  air  defense 
system  has  to  be  assumed.  Survival  Probabilities  for  rotary  wing  aircraft  against 
medium  altitude  threats,  such  as  the  SA-6,  need  to  assume  the  presence  of  low 
altitude  anti-aircraft  weapon  systems,  even  though  they  are  not  normally 
specified.  This  is  a  valid  tactical  assumption  that  was  not  considered  in  the 
development  of  the  decision  aid  and  is  required  when  allowing  rotary  assets  to 
be  considered  in  a  feasible  solution  set. 
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3.  Interoperability 

The  decision  aid  is  currently  able  to  receive  and  parse  an  ATO  from  the 
current  system  of  record,  the  Theater  Battle  Management  Core  System 
(TBMCS).  However,  it  is  recommended  that  the  decision  aid  be  further 
enhanced  to  receive  near  real  time  updates  in  regards  the  aircraft  status. 

4.  Alternative  Solution  Proposal 

The  decision  aid  currently  proposes  only  one  solution  option  per  TST 
event.  It  is  recommended  that  the  decision  aid  provide  at  least  three  to  meet  the 
needs  of  the  tactical  commander,  as  recommended  by  the  instructors  at  Marine 
Aviation  Weapons  and  Tactics  Squadron  One. 

5.  Software  Reliability 

The  duration  of  each  exercise  for  the  test  and  evaluation  varied  from  three 
to  eight  hours  in  length.  Normal  tactical  operations  run  at  least  24  hours  in 
length.  Although  no  software  interruptions  were  encountered  during  the  test  and 
evaluation,  it  is  recommended  that  more  testing  be  done  to  support  the 
conclusion  of  reliability. 

C.  FURTHER  RESEARCH 

The  following  sections  provide  recommendations  for  additional  research 
stemming  from  this  thesis. 

1.  ATO  Development 

The  decision  aid  has  proven  effective  in  re-optimizing  the  current  ATO 
due  to  an  immediate  change  in  the  tactical  situation.  Testing  revealed  that 
employing  an  optimization  tool  during  ATO  development,  as  well  as,  execution 
could  have  a  similar  impact  in  solution  time  and  accuracy  of  aircraft  assignments. 

2.  Weather  Effects 

To  consideration  of  weather  effects  on  weapon  selection  is  not  a 
consideration  of  the  current  decision  aid.  The  incorporation  of  this  type  of  effect 
will  further  improve  the  accuracy  of  each  asset  assignment. 
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3.  Collateral  Damage  Estimation 

A  TST  can  emerge  anywhere  on  the  battlefield.  However,  significant 
concern  is  given  to  the  TST  that  emerges  in  an  urban  environment  where  the 
cost  of  attacking  a  TST  can  manifest  itself  in  civilian  casualties.  The  ability  to 
minimize  the  effects  of  asset  selection  on  factors  outside  of  the  target  provides  a 
current  and  tactically  relevant  subject  for  further  research. 
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1.0  INTRODUCTION 

1.1  Purpose 

This  tactical  evaluation  will  assess  and  compare  alternative  methods  in 
determining  real  time  re-assignment  of  aircraft  to  time  sensitive  or  high  priority 
targets  through  either  automated  or  manual  means.  These  will  include: 

1.  Current  manual  assignment  procedures  based  on  military  judgment  and 
experience  taught  in  the  Weapons  and  Tactics  Instructors  (WTI)  Course. 

2.  Currently  Available  Software  Packages  employed  by  operating  forces. 

3.  The  Rapid  Asset  Pairing  Tool  (RAPT),  a  genetic-algorithm  based  optimization 
tool  under  development  by  SPAWAR. 

4.  An  integer-programming  based  optimization  tool  developed  by  Major  Davi 
Castro  and  Professor  Richard  E.  Rosenthal  of  the  Operations  Research 
Department,  Naval  Postgraduate  School  (NPS). 

1.2  System  Description 

This  tactical  evaluation  will  be  accomplished  within  the  Marine  Tactical  Air 
Command  Center  (TACC)  present  during  the  Weapons  Tactics  Instructors  (WTI) 
Course.  Assignment  of  aircraft  to  reactive  targets  by  employing  manual  methods 
will  be  conducted  by  the  students  of  the  Weapons  and  Tactics  Instructor’s 
Course.  These  students  are  experienced  officers  in  the  aviation  community 
going  through  instructor  qualifications.  These  students  will  be  aided  by  the 
employment  of  current  command  and  control  systems  that  may  or  mat  not 
employ  any  optimization  techniques.  RAPT,  an  optimization  tool  designed  by 
SPAWAR,  has  the  capability  to  be  integrated  into  the  exercise  command  and 
control  architecture  and  receive  near  real  time  updates  concerning  the  friendly 
and  enemy  situation.  These  updates  include  but  are  not  limited  to  aircraft 
availability,  time  on  station,  and  ordnance  load  out.  If  this  level  of  integration  is 
unable  to  be  accomplished,  RAPT  as  well  as  a  similar  tool  developed  by  NPS 
can  and  will  operate  independently  by  relying  on  manual  inputs  by  the  test 
personnel.  The  ultimate  goal  of  this  evaluation  is  to  determine  the  tactical  quality 
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of  each  systems  output  compared  to  that  of  an  experienced  operator  using 
current  methods.  A  reliance  on  manual  inputs  of  the  system  will  merely  identify  a 
need  for  further  development  in  integration  of  the  system. 

2.0  Mission  Need  and  Operationai  Requirement 
2.1  Mission  Need 

The  following  mission  need  statement  outlined  in  the  Joint  Mission 
Planning  System  (JMPS)  Operational  Requirements  Document  (ORD)  has  been 
the  guiding  element  in  developing  the  Rapid  Asset  Pairing  Tool  (RAPT)  by 
SPAWAR.  To  achieve  the  Joint  Vision  2020  (JV2020)  operational  concepts  of 
dominant  maneuver  and  precision  engagement,  future  systems  must  be 
designed  to  streamline  many  areas  of  the  targeting/decision  cycle,  including 
receipt  of  initial  tasking/Commander’s  Intent,  gathering/disseminating 
intelligence,  joint/unit-level  detailed  planning,  dynamic  mid-mission  re-tasking, 
and  mission  effectiveness  assessment.  Factors  driving  the  need  for  streamlined 
and  shorter  mission  planning  cycle  times  include  mobility  of  targets,  the 
complex/dynamic  environment  of  large  numbers  of  joint  assets,  and  greater 
dependency  of  modern  aircraft/precision  weapons  on  large  quantities  of 
intelligence  data.  In  this  highly  dynamic  battlespace,  successful  execution  will 
depend  on  a  planning  system  that  provides  seamless  and  efficient  access  to  all 
required  data  sources,  including  those  maintained  on  systems  operating  at 
differing  classification  levels,  and  manipulates  the  data  to  provide  information, 
data  products  and  options  to  planners  in  user  friendly  formats.  JV  2020 
envisions  expanding  roles  for  multinational  and  interagency  partners,  which  will 
require  collaborative  planning  capabilities,  technological  compatibility/ 
interoperability,  and  mechanisms  for  efficient  information  sharing. 

The  focus  in  the  development  of  RAPT  has  been  on  three  key  elements 
identified  within  the  mission  need  statement.  They  are:  “to  streamline  many 
areas  of  the  targeting/decision  cycle,  dynamic  mid-mission  re-tasking,  and 
mission  effectiveness  assessment.”  [JMPS  ORD,  2002] 
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2.2  Operational  Requirement 

The  following  is  a  list  of  user  defined  operational  requirements  to  be 
used  in  the  evaluation  of  the  systems  developed  by  SPAWAR  and  the  Naval 
Postgraduate  School. 


System  software  will  be  able  to  be  mounted  on  current  deployable  computing 
hardware. 

System  will  be  able  to  operate  continually  without  interruption  for  72  hours. 

System  will  be  able  to  determine  most  effective  asset  to  target  pairing  to  be  used 
given  an  aircraft  type,  ordnance  load,  and  associated  threat  type. 

System  will  be  able  determine  overall  risk  to  assigned  aircraft  given  a  specific  threat. 

System  will  be  able  to  receive  an  ATO  transmitted  by  TBMCS. 

Aircraft  assignment  will  appear  as  a  system  output  in  a  timely  manner. 

System  will  be  able  to  measure  current  impact  of  decision  to  the  Air  Tasking  Order. 

System  will  be  intuitive  and  require  minimal  training. 

3.0  SCOPE  OF  THE  EVALUATION 
3.1  Criticai  Technicai  Parameters 

The  following  is  a  list  of  user  defined  Critical  Technical  Parameters  to  be 

used  in  the  evaluation  of  the  systems  developed  by  SPAWAR  and  the  Naval 

Postgraduate  School. 

•  At  a  minimum  the  system  will  be  able  to  receive  data  from  TBMCS  and  the 

•  Joint  Mission  Planning  System  (JMPS). 

•  Proposed  assignment  will  have  a  probability  of  destruction  >  prescribed 
destruction  criteria  for  the  given  target. 

•  System  will  determine  either  a  probability  of  survival  or  expected  attrition  rate  of 
assets  assigned  against  the  given  target. 

•  System  output  will  provide  at  a  minimum,  3  proposed  target  assignments,  assets 
permitting. 

•  Proposed  aircraft/target  assignments  will  be  completed  within  one  minute. 

•  System  can  parse  an  ATO  with  a  minimum  100  sorties  and  maximum  5000. 

•  System  will  display  at  minimum: 

o  Probability  of  Destruction 
o  Risk  Assessment  or  Expected  Value  of  Attrition 

•  Persistence  measurement  will  convey  number  of  changes  or  percentage  impact 
to  the  ATO 

•  Current  Mission  Number 

•  Ordnance  Load-out 

•  Previously  Assigned  Target  and  its  priority 

•  New  Target,  its  Priority,  and  Associated  threat 
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3.2  Critical  Operational  Issues 

COI  1.0  -  Target  Selection:  Does  the  system  properly  assess 
prioritization  of  targets  to  ensure  mission  accomplishment  of  higher  priority 
targets  before  lower  priority  targets?  Targets  are  prioritized  numerically 
beginning  with  one,  the  highest  priority  target.  Given  a  situation  where  all  assets 
are  currently  assigned  the  potential  for  a  target  not  being  struck  exists.  Each 
decision  tool  must  be  able  to  accurately  assign  assets  so  that  the  higher  priority 
targets  are  paired  with  assets. 

MOE  1.1  -  Accurate  target  selection 

MOP  1.1.1  -  Percent  properly  selected 

DR  1.1. 1.1  -  Total  Missions  assigned. 

DR  1.1. 1.2  -  Number  of  Missions  not  assigned  with  a  higher  priority  by 
RAPT  or  NPS  tool. 

DR  1.1. 1.3  -  Number  of  Missions  not  assigned  by  DM  without  the  aid 
of  an  automated  tool 

COI  2.0  -  Asset  Availability;  Does  the  system  properly  determine  the 
assets  that  are  available  for  a  particular  target  in  its  analysis?  Available  assets 
are  those  that  have  not  been  cancelled  on  the  ATO  for  mechanical  failures  or 
those  that  have  been  sent  on  to  original  target. 

MOE  2.1  -  Asset  consideration  accuracy 

MOP  2.1.1  -  Percent  available  considered.  Determined  by  the  number  of 
assignments  made  that  consisted  of  available  missions.  This  does  not  include 
missions  that  have  already  proceeded  to  original  target,  or  that  have  been 
canceled  due  to  maintenance. 

DR  2.1 .1 .1  -  Actual  number  of  aircraft  available  and  capable 

DR  2. 1.1. 2  -  Number  assigned  that  were  considered  available  and 
capable  by  the  NPS  or  RAPT  tool. 

DR  2. 1.1. 3  -  Number  assigned  that  were  considered  available  and 
capable  by  the  DM  without  the  aid  of  an  automated  tool. 
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COI  3.0  -  Target  Asset  Pairing:  Does  the  system  ensure  aircraft  are 
properly  reassigned  to  targets  they  can  destroy  the  target?  Each  mission’s 
ordnance  load  against  a  given  target  can  be  computed  by  the  decision  aids  to 
determine  a  level  of  destruction.  We  wish  to  determine  if  this  destruction  level  is 
equal  to  greater  than  the  destruction  criteria  required  for  each  target  assigned  an 
asset  by  the  decision  aid.  Assets  assigned  that  meet  the  desired  destruction 
criteria  will  be  considered  a  correct  assignment. 

MOE  3.1  -  Allocation  Plan 

MOP  3.1.1  -  Percent  Correct  Assignments 

DR  3. 1.1.1-  Number  of  assignments  made  by  NPS  or  Rapt  tool  that 
could  not  destroy  targets 

DR  3. 1.1.1-  Number  of  assignments  made  by  DM  without  the  aid  of  an 
automated  tool  that  could  not  destroy  targets 

DR  3. 1.1.2-  Total  Number  of  assignments 

COI  4.0  -  Mission  Risk  Assessment;  Does  the  system  accurately  assess 
risk  in  its  reassignments?  Given  a  threat  associated  with  a  given  target  each 
decision  aid  is  expected  to  compute  a  level  of  risk  to  the  assets  proposed  for 
assignment. 

MOE  4.1  -  Rate  at  which  is  accurately  assessed 

MOP  4.1.1  -  Percent  properly  assessed.  Assessment  to  be  done  by 
qualified  instructors  on  the  MAWTS-1  Staff. 

DR  4.1. 1.1  -  Computed  risk  assessment  for  each  target/aircraft  pair 
DR  4.1. 1. 2  -  Risk  Assessment  by  DM. 

MOE  4.2-  Proper  SEAD  recommendation  based  on  Risk  assessment 

MOP  4.2.1-  Percent  SEAD  assignments  made  when  required 

DR  4.2.1-  Actual  SEAD  assignments  made  when  required  by  RAPT 
or  NPS  tool. 

DR  4.2.1-  Actual  SEAD  assignments  made  when  required  by  DM. 

DR  4.2.2-  Number  of  SEAD  assignments  requiring  SEAD  support. 


85 


COI  5.0  -  Persistence  in  ATO:  Does  the  system  minimize  the  number  of 
changes  to  the  ATO  to  achieve  mission  accomplishment?  Each  decision  aid  is 
expected  to  select  an  asset  for  assignment  that  is  affords  the  least  impact  to 
existing  assignments. 

MOE  5.1  -  ATO  Persistence. 

MOP  5.1.1-  Number  of  changes  to  the  current  ATO 

DR  5. 1.1.1  -  Number  changes  required  for  re-tasking  using  RAPT  or 
NPS  tool  for  each  new  target  input. 

DR  5.1. 1.2  -  Number  changes  required  for  re-tasking  by  DM  without 
the  aid  of  an  automated  tool  for  each  new  target  input. 

COI  6.0  -  Timeiiness:  Is  the  system  fast  enough  to  be  effective  compared 
to  current  methods? 

MOE  6.1  -  Decision  Speed 

MOP  6.1.1  -  Average  time  to  output  recommendation  to  the  DM 

DR  6. 1.1.1  -  Time  from  receipt  of  target,  to  solution  recommended  to 
DM  with  the  NPS  or  RAPT  tools 

DR  6. 1.1. 2  -  Time  from  receipt  of  target,  to  solution  recommended  to 
DM  without  the  decision  aid. 

COI  7.0  -  Options;  Does  the  system  provide  the  decision  maker  with 
multiple  alternative  solutions  for  mission  re-tasking  when  available? 

MOE  7.1  -  Multiple  alternative  generation 

MOP  7.1.1  -  Number  of  provided  to  available  alternatives 

DR  7.1. 1.1  -  Number  of  alternatives  provided  by  the  NPS  or  RAPT  tool 
DR  7.1. 1.2  -  Number  of  alternatives  provided  to  DM  without  the  aid  of 
an  automated  tool. 

COI  8.0  -  Tactical  Accuracy;  Does  the  system  output  provide  tactically 
acceptable  assignments  to  ensure  the  ability  of  assets  reassigned  to  accomplish 
the  mission?  The  output  for  each  assignment  will  be  assessed  to  determine  if 
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the  proposed  assignments  are  reasonable.  This  is  not  assessment  of  optimality 
but  a  determination  of  whether  or  not  the  proposed  solution  is  one  of  many 
possibilities. 

MOE  8.1  -  Probability  of  correct  output  is  tactically  acceptable 

MOP  8.1.1  -  Percent  output  is  tactically  unacceptable 
DR  8.1 .1 .1  -  Number  of  acceptable  assignments 
DR  8.1 .1 .2-  Number  of  assignments  made 

COI  9.0  -  Interoperability;  Does  the  system  operate  correctly  with  the 
current  C4I  architecture? 

MOE  9.1  -  Probability  of  properly  receive  the  ATO 

MOP  9.1.1  -  Percent  of  ATO  received  properly 
DR  9. 1.1.1  -  Number  of  ATOs  received 
DR  9. 1.1. 2  -  Number  of  ATOs  sent 

COI  10.0  -  Software  Reliability:  Can  the  automated  tools  operate 
continuously  without  interruption  or  failure  for  a  prolonged  period  of  time? 

MOE  10.1  -  Reliability 

MOP  10.1.1  -  Mean  time  before  failure 

DR  10.1.1.1  -  Time  between  each  failure  for  the  RAPT  and  NPS  tool 

COI  11.0  -  Human  Factors:  Can  the  typical  user  enter  inputs  and 
understand  outputs  in  an  efficient,  intuitive  manner  with  minimal  training? 

MOE  11.1  -  Display 

MOP  11.1.1  -  Percent  operators  satisfied  with  the  display. 

DR  11.1.1.1  -  Number  satisfied  w/  Probability  of  destruction  display 
DR  1 1 .1 .1 .2  -  Number  satisfied  w/  Risk  Assessment  display 
DR  11.1.1.3  -  Number  satisfied  w/  Persistence  measurement  display 
DR  11.1.1.4  -  Number  satisfied  w/  Current  Mission  Number  display 
DR  11.1.1.5  -  Number  satisfied  w/  Ordnance  Capability  display 
DR  11.1.1.6  -  Number  satisfied  w/  display  regarding  the  previously 
assigned  target,  its  priority,  and  associated  threat. 
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DR  11.1.1.7-  Number  satisfied  w/the  display  regarding  the  New  Target 
and  its  priority  and  associated  threat 

DR  11.1.1.8  -  Total  number  of  Operators 


Target 

Selection 


Asset 

Availability 

Target  Asset 
Pairing 


Mission  Risk 
Assessment 


Persistence  in 
ATO 


Timeliness 


Options 


Tactical 

Accuracy 

Software 

Reliability 

Interoperability 


Human 

Factors 


Table  1.  Test  Objective  Matrix 


_ Test  Objectives  and  Sub-Objectives _ 


To  determine  if  system  properly  selects  targets 
for  assignment  to  ensure  mission  accomplishment  of 
higher  priority  targets  before  lower  priority  targets. 
-Accuracy 

-System  consistency  with  manual  methods _ 

To  evaluate  if  the  system  properly  determines 
the  assets  that  are  available  for  a  particular  target. 

-Proportion  considered _ 

To  assess  if  the  system  properly  reassigns 
aircraft  to  targets  they  can  destroy  at  a  certain  level  of 
confidence. 

-Proportion  correctly  calculated 

-Probability  of  kill  for  assigned  aircraft _ 

To  determine  if  the  system  avoids  undue  risk  in 
its  reassignments. 

-Proportion  properly  assessed 

-SEAD  Assignment  recommendation _ 

To  determine  if  the  system  minimizes  the 
number  of  changes  to  the  ATO  to  achieve  mission 
accomplishment. 

-Proportion  properly  predicted 

-Persistence  ratio _ 

To  assess  if  the  system  is  fast  enough  to  be 
effective  compared  to  current  methods. 

-Decision  time _ 

To  determine  if  the  system  provides  the  decision 
maker  with  multiple  alternative  solutions  for  mission 
retasking,  when  available. 

-  Provided  to  available  alternatives _ 

To  determine  if  system  recommendations  for 
asset  pairing  is  tactically  accurate. 

To  determine  if  the  automated  tools  can  operate 
for  continuously  without  failure. 

-Mean  time  before  failure _ 

To  determine  if  the  system  operates  correctly 
with  the  current  C4I  architecture. 

-  ATOs  received  correctly 

-  Updates  received 

To  evaluate  if  the  typical  user  can  enter  inputs 
and  understand  outputs  in  an  efficient,  intuitive  manner 
with  minimal  training. 

-Displays 
-Training  time 
-Interpretation  errors 
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3.4  General  Test  Operations,  Test  Vehicles,  and  Scenario  Overview 
3.4.1  Generai  Test  Operations 

The  evaluation  will  be  conducted  during  the  Weapons  Tactics 
Instructor’s  Course  (WTI)  2-04.  The  course  is  executed  by  Marine  Aviation 
Weapons  and  Tactics  Squadron  One  (MAWTS-1)  located  at  Marine  Corps  Air 
Station  (MCAS)  Yuma,  Arizona.  The  tools  and  methods  to  be  evaluated  will 
be  located  in  the  Marine  Tactical  Air  Command  Center  (TACC).  The  TACC  is 
the  senior  aviation  command  and  control  agency  for  the  Marine  Corps  and  is 
ultimately  responsible  for  the  conduct  of  the  air  war,  to  include  real  time  re¬ 
tasking  of  aviation  assets.  WTI  is  designed  to  train  and  evaluate  the  Marine 
Corps  experienced  Aviators,  Command  and  Control  Officers,  and  Aviation 
Ground  Support  Officers  in  the  conduct  of  their  assigned  specialties.  Once 
training  is  complete  the  officers  and  enlisted  personnel  are  sent  back  to  their 
units  to  become  the  resident  experts  for  the  unit.  Within  the  Marine  Corps 
aviation  community,  WTI  has  been  referred  to  as  a  graduate  level  course  for 
tactics. 

The  execution  of  a  WTI  course  presents  a  unique  opportunity  to 
evaluate  not  just  our  Marines,  but  the  tools  developed  to  support  the  conduct 
of  their  mission.  The  course  is  divided  into  two  phases;  Academic  and  Flight. 
The  flight  phase  will  provide  the  vehicle  for  this  evaluation. 

During  the  flight  phase  the  evaluation  will  focus  on  only  the  final  two 
separate  flight  phase  evolutions.  The  MACCS  Integrated  Simulated  Training 
Exercise  (MISTEX),  and  the  Final  Exercise  (FINEX)  will  comprise  the  specific 
test  vehicles  for  the  evaluation. 

MISTEX  is  a  static  exercise  consisting  of  the  entire  Marine  Aviation 
Command  and  Control  System  in  the  field  executing  a  realistic,  low  intensity, 
tactical  scenario  over  a  two-hour  period.  This  MISTEX  scenario  is  executed 
twice  over  a  two-day  period  to  ensure  the  MACCS  is  fully  operational  prior  to 
the  beginning  of  flight  operations. 
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FINEX  is  the  culminating  exercise  of  the  WTI  course.  It  is  highly 
intensive  and  demonstrates  all  six  functions  of  Marine  Aviation.  These  six 
functions  include  Control  of  Aircraft  and  Missiles,  Anti-Air  Warfare,  Offensive 
Air  Support,  Assault  Support,  Air  Reconnaissance,  and  Electronic  Warfare. 
Functions  are  driven  by  a  tactical  scenario  where  command  and  control 
officers  are  exercising  their  delegated  authority  in  the  direction  and  control  of 
aircraft.  The  determination  of  whether  or  not  our  tools  can  adequately 
support  the  decision  makers  in  this  type  of  operation  will  be  assessed  during 
this  evolution. 

3.4.2  Scenario  Overview 

The  scenario,  for  the  purposes  of  the  evaluation  consists  of  offensive  and 
defensive  combat  operations  involving  ground  and  air  forces  against  a  heavy 
mechanized  division  in  a  desert  environment.  The  enemy  air  capabilities  consist 
of  two  airforce  groups  with  a  mixture  reconnaissance,  anti-air,  air  to  ground,  and 
support  capabilities.  In  addition,  a  major  consideration  of  the  friendly  forces  will 
be  the  enemy’s  surface-to-surface,  and  surface  to  air  missile  threat.  The  phase  of 
the  operation  in  which  our  systems  will  be  evaluated  will  primarily  involve 
offensive  air  support  and  assault  support  operations.  The  general  description  of 
the  enemies  capabilities  provide  the  source  of  targets  and  threats  for  our 
decision  aids  to  be  evaluated  against.  The  enemy  forces  will  be  simulated  on  the 
ground  either  through  the  use  of  static  targets  or  electronic  threat  emitters. 
Friendly  aviation  support,  command  and  control,  and  air  defense  forces  used  in 
the  execution  of  this  scenario  will  be  live.  Friendly  ground  combat  forces  are 
represented  by  one  USMC  infantry  company  and  one  USMC  Artillery  battery. 
There  will  be  no  notional/simulated  friendly  aircraft  during  the  execution  of  the 
flight  phase. 

3.5  Instrumentation  Requirements 

Assets  required  that  are  outside  of  those  used  to  conduct  the  WTI  course 
will  be  provided  by  SPAWAR.  These  requirements  are  limited  to  two  Laptop 
computers  capable  of  running  the  software  to  be  evaluated. 
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3.6 


Limitations  and  Scope  of  Test 

The  following  subsections  and  table  provides  in  detail,  the  scope  and 
limitations  of  the  test  to  be  conducted.  A  static  profile  as  indicated  on  the  table 
denotes  no  actual  aircraft  being  flown.  In  this  instance  a  simulated  air  picture  will 
be  provided  to  the  command  and  control  agency.  A  flight  profile  denotes  the  use 
of  live  aircraft  and  ordnance  used  during  the  evaluation. 

3.6.1  Phase  I:  Pre-Evaluation 

Testing  will  be  conducted  2-3  April  2004  to  ensure  equipment  functionality 
and  feasibility  of  operations.  This  testing  will  be  conducted  within  the  TACC 
located  at  MCAS  Yuma.  We  will  also  use  this  phase  to  record  and  review  initial 
data. 

3.6.2  Phase  II:  Final  Evaluation 

Testing  will  commence  on  13  April  and  conclude  on  16  April  2004.  This 
testing  will  ensure  interoperability  and  evaluate  the  different  decision  tool’s 
command  and  control  effectiveness.  Ultimately  this  will  provide  the  basis  for  our 
comparative  analysis  of  the  two  tools  and  the  manual  methods  currently 
employed  by  the  decision  makers. 

Table  2.  T&E  Phase  Summary 


Phases 

Date 

Time 

Evolution 

Profile 

Phase  1:  Pre- 
Evaluation 

2-3  April 

TBD 

MISTEX 

Static 

Phase  II:  Final 
Evaluation 

13-16  April 

TBD 

FINEX 

Flight 

4.0  Operational  Effectiveness 
Test  E-1.  Target  Selection. 

Objective:  To  assess  whether  or  not  the  methods  evaluated  maintain 
target  priorities  when  reassigning  assets. 

Procedure:  Each  target  assigned  attack  assets  on  the  existing  ATO  have 

priorities.  A  target  with  the  priority  of  one  is  the  highest  priority.  Each 

immediate  time  sensitive  or  high  priority  target  is  also  assigned  a  similar 
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priority.  As  each  immediate  target  appears  during  the  exercise  the  decision 
aids  will  be  expected  to  assign  assets  based  on  these  priorities.  If  the  results 
reveal  that  a  previously  assigned  target  that  is  not  paired  with  an  asset  after 
running  the  decision  aids  has  a  higher  priority  than  those  assigned  to  assets  it 
will  be  counted.  Operators  will  annotate  on  the  effectiveness  data  sheet  the 
number  of  targets  that  are  of  higher  priority  than  those  assigned  assets. 

Data  Analysis:  Accurate  target  selection  will  be  quantitative  in  nature. 
The  analysis  will  be  based  on  the  number  of  higher  priority  targets  that  do  not 
have  an  asset  assigned  per  the  total  number  of  assets  assigned  to  targets. 
Operators  will  annotate  on  the  data  collection  sheet  (Appendix  B)  the  number 
of  targets  with  no  assets  assigned  that  have  a  higher  priority  than  the  targets 
assigned  assets.  During  each  run  of  the  decision  tool  the  operator  will  also 
annotate  the  number  total  number  assignments  that  were  considered  by  each 
decision  aid. 

Test  E-2.  Asset  Availability. 

Objective:  To  assess  whether  or  not  the  methods  under  evaluation 

select  assets  for  reassignment  that  are  currently  available. 

Procedure:  The  operator  will  record  the  number  of  assignments 

suggested  by  the  decision  aid  that  are  unavailable  and  annotate  that  on  the 
effectiveness  data  sheet  (Appendix  B).  For  example,  assets  that  have  been 
identified  to  have  mechanical  problems,  or  assets  that  have  already 
proceeded  to  assigned  targets.  This  effectiveness  test  does  not  take  into  lack 
of  proper  reporting  by  the  exercise  force.  Regardless  of  the  reason  for  the 
decision  aids  consideration  of  unavailable  assets  the  operator  will  annotate 
the  number  of  erroneous  assignments  and  the  total  number  of  assignments. 
The  remarks  section  on  the  data  sheet  will  be  used  to  provide  reasons  for  the 
decision  tools  assignment  if  erroneous. 

Data  Analysis:  Asset  Availability  will  be  assessed  quantitatively.  A 
higher  ratio  of  miss-assignments  will  aid  in  identifying  a  system  problem  in 
recognition  of  available  assets.  The  reliability  of  this  measurement  will 
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depend  on  the  timeliness  and  accuracy  of  information  about  current  aircraft 
status.  Each  method  will  be  compared  to  determine  which  has  a  lower  rate  of 
assignment  of  assets  that  are  not  available.  A  higher  rate  of  assignment  of 
assets  not  available  will  help  identify  a  problem  with  either  receiving 
automated  updates  or  manually  updating  asset  availability. 

Test  E-3.  Target  Asset  Pairing. 

Objective:  To  assess  whether  or  not  the  methods  under  evaluation 
properly  reassign  assets  to  target  that  have  the  ability  to  meet  desired 
destruction  criteria  provided  by  the  component  commander. 

Procedure:  Each  decision  aid  computes  the  probability  of  destruction 
against  a  particular  target  based  on  the  assets  ordnance  capability.  Each 
decision  aid  pre-calculates  these  values.  The  desired  destruction  criteria  is 
provided  by  the  destruction  criteria  matrix  developed  by  the  exercise  force. 
Operators  will  evaluate  the  number  of  missions  reassigned  to  the  immediate 
target  that  have  a  probability  of  destruction  that  is  less  than  the  stated 
destruction  criteria.  The  results  will  be  recorded  by  either  a  yes  the 
destruction  criteria  is  met  by  the  results  of  the  decision  aid  or  no  it  is  not  on 
the  data  collection  sheet  (Appendix  B).  Additionally,  the  destruction  level 
attained  by  the  proposed  assignment  well  be  annotated  on  the  data  collection 
sheet. 

Data  Analysis:  This  measure  is  quantitative  in  nature.  Each  method  either 
automated  or  manual  will  be  compared  to  determine  which  achieves  a  lower  rate 
of  inadequate  assignment.  The  total  number  of  assignments  that  are  below  the 
probability  of  destruction  will  be  annotated  on  the  data  collection  sheet.  A 
proposed  assignment  that  does  not  meet  the  destruction  criteria  will  indicate 
either  an  error  in  the  method  for  calculating  weapons  effectiveness  against  a 
given  target  type  or  lack  of  availability  of  assets  necessary  to  destroy  a  given 
target.  In  the  case  of  the  latter  we  will  be  able  to  assess  the  ability  of  the 
decision  aids  to  assign  assets  that  can  achieve  the  best  results  possible. 
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Test  E-4.  Mission  Risk  Assessment. 

Objective:  To  determine  whether  or  not  each  method  evaluated  accurately 
assesses  and  takes  action  to  mitigate  the  risk  from  associated  surface  to  air 
threats. 

Procedure:  The  proper  assessment  of  risk  will  be  determined  post  exercise 
by  evaluating  each  assignment  to  determine  an  accurate  risk  assessment. 
Operators  will  maintain  a  record  of  each  assignment’s  risk  assessment  whether 
that  is  in  the  form  of  a  probability  aircraft  loss  or  expected  value  of  attrition. 
Based  on  this  assessment  operators  will  determine  if  the  system  or  decision 
maker  assigned  BEAD  assets  when  available  to  minimize  risk.  Operators  will 
annotate  risk  assessment  on  the  effectiveness  data  sheet  (Appendix  B)  provided. 
The  assignment  of  BEAD  will  be  annotated  with  either  a  yes  or  a  no  on  the  data 
collection  sheet  (Appendix  B). 

Data  Analysis:  Two  elements  will  be  analyzed.  First,  the  number  of  proper 
risk  assessments  for  each  aircraft  will  be  counted  to  determine  a  percentage  of 
proper  assessment.  Becond,  a  comparative  analysis  will  be  conducted  to 
determine  which  method  produced  the  assignments  with  minimal  risk  and 
assigned  BEAD  when  required.  A  failure  to  properly  assess  risk  to  aircrew  given 
a  surface-to-air  threat  could  point  to  a  problem  in  the  values  used  in  the 
probability  of  kill  for  each  threat  or  a  problem  in  the  formulation  of  risk. 

Test  E-5.  Persistence. 

Objective:  To  evaluate  whether  each  method  attempts  to  minimize  the 
impact  of  the  reassignment  to  the  current  ATO. 

Procedure:  Each  ATO  can  be  considered  a  previously  optimized  assignment 
plan.  A  immediate  target  necessitates  the  re-optimization  of  the  ATO.  Each 
method  when  conducting  the  reassignment  takes  steps  to  insure  that  as  much  of 
the  current  ATO  as  possible  stays  intact  while  assigning  an  asset  to  the 
immediate  target.  The  number  of  changes  the  ATO  can  be  reflected  in  the 
number  of  calls  that  have  to  be  made  to  affect  the  new  assignment. 
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Data  Analysis:  A  comparative  analysis  will  be  conducted  to  determine  which 
of  the  methods  effectively  minimize  the  number  of  changes  to  the  current  ATO 
after  conducting  the  reassignment  of  assets.  Time  to  act  on  the  proposals 
provided  by  the  decision  aids  determines  the  usefulness  of  the  proposal.  Too 
many  changes  in  a  short  period  of  time  will  indicate  a  solution,  although 
technically  superior,  that  may  not  be  feasible. 

Test  E-6.  Timeliness. 

Objective:  To  determine  whether  or  not  the  methods  under  evaluation  can 
produce  courses  of  action  for  the  decision  maker  in  a  timely  enough  manner  to 
assist  in  making  a  decision. 

Procedure:  Throughout  the  conduct  of  the  exercise  immediate  targets  will 
appear  that  require  assignment  of  assets.  At  each  appearance  the  operator  will 
annotate  on  the  data  collection  sheet  the  time.  Target  information  will  be  entered 
into  the  appropriate  system  to  await  results.  Once  the  results  are  displayed  the 
time  will  then  be  annotated  on  the  data  collection  sheets  (Appendix  B).  From  this 
information  we  will  determine  the  total  time  to  process  the  information  and  make 
a  recommendation  to  the  decision  maker.  We  will  then  determine  the  average 
time  for  each  method. 

Data  Analysis:  The  analysis  is  quantitative  in  nature.  Each  methods  time 
will  be  compared  to  determine  the  method  that  can  produce  results  in  the 
shortest  period  of  time.  Each  failure  will  be  categorized  into  one  of  three  types: 
software,  hardware,  or  network  failure  and  annotated  on  the  reliability  data 
collection  sheet.  (Appendix  C) 

Test  E-7.  Options. 

Objective:  To  evaluate  whether  the  methods  provide  the  decision  maker 
with  more  than  one  course  of  action. 

Procedure:  For  each  implementation  of  the  decision  tools  the  number  of 
options  provided  by  the  decision  maker  will  be  counted  by  the  operator  and 
annotated  on  the  data  sheet  (Appendix  B).  For  every  immediate  target  there 
exists  more  than  one  possible  solution. 
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Data  Analysis:  The  average  number  of  options  provided  by  the  decision 
aids  will  be  calculated  and  compared  between  each  of  the  methods. 

Test  E-8.  Tactical  Accuracy. 

Objective:  To  evaluate  whether  or  not  the  output  by  the  automated 

decision  tools  provide  results  that  can  be  considered  tactically  acceptable.  This 
is  one  of  the  primary  purposes  of  the  evaluation. 

Procedure:  MAWTS-1  Instructors  present  during  the  evaluation  will  examine 
each  data  sheet. 

The  information  on  the  data  sheets  represents  the  proposed  solution  for  each 
immediate  target  that  presents  itself  during  the  exercise.  Their  comments  will  be 
annotated  on  the  effectiveness  data  sheet  provided  (Appendix  B). 

Data  Analysis:  This  analysis  is  subjective  in  nature.  An  overall  assessment 
will  be  made  as  to  the  accuracy  of  the  proposed  assignments  for  each  individual 
method. 

5.0  Operational  Suitability 

Test  S-1.  Software  Reiiabiiity. 

Objective:  To  determine  if  the  automated  tools  can  operate  continuously 
without  interruption  throughout  the  entire  tactical  evolution. 

Procedure:  The  tactical  evolutions  will  run  approximately  eight  hours. 

During  this  time  the  number  of  system  failures  will  be  annotated  on  the  reliability 
data  sheet  provided.  Each  failure  will  be  noted  with  the  time  of  failure  and  time  of 
restart. 

Data  Analysis:  The  analysis  is  both  objective  and  subjective.  Quantitatively 
we  will  assess  the  mean  time  before  failure.  Subjectively  we  will  determine  if  it  is 
a  fault  of  the  hardware,  software  or  exercise  architecture. 

Test  S-2.  Interoperability. 

Objective:  To  determine  if  each  system  under  evaluation  is  able  to 
electronically  receive  the  ATO. 
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Procedure:  Prior  to  the  start  of  each  exercise  an  ATO  will  be  disseminated 
by  MAWTS-1 .  The  ability  to  receive  the  ATO  will  be  identified  with  either  a 
yes  or  no  on  the  data  sheet. 

Data  Analysis:  The  assessment  will  be  quantitative  in  nature.  We  will 
assess  which  method  on  average  is  able  to  receive  the  ATO.  Interoperability 
problems  will  be  recorded  as  they  occur. 

Test  S-3.  Human  Factors. 

Objective:  Can  the  display  be  easily  interpreted  with  respect  to  its  inputs  and 
outputs? 

Procedure:  Using  the  suitability  questionnaire  a  perception  of  how  well  the 
interface  display  for  each  system  explains  the  results  to  the  user.  Each  aspect 
identified  in  the  data  requirements  will  be  assessed. 

Data  Analysis:  The  analysis  is  subjective  in  nature.  Each  method  will  be 
assessed  individually  to  determine  usability. 
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LOGISTICS  AND  SUPPORT  REQUIREMENTS 


1.0  MAWTS-1  PERSONNEL  REQUIREMENTS 

1.1  Aviation  Development,  Tactics,  and  Evaluation  (ADT&E)  Department 

1.1.1  Coordinating  Officer 

Coordinate  the  efforts  of  all  T&E  participants  throughout  the  duration  of 
the  T&E  process. 

Co-develop  T&E  plan. 

Co-develop  operator  checklists  and  questionnaires. 

Provide  assistance  in  developing  the  T&E  final  report. 

1.1.2  Aviation  Operations  Analyst 

Provide  analytical  assistance  to  support  the  T&E. 

Provide  assistance  to  the  Project  Action  Officer  in  writing  the  final  report. 

1.2  Command,  Control,  and  Communications  (C3)  Department 
1.2.1  Project  Action  Officer 

Coordinate  the  efforts  of  all  T&E  participants  throughout  the  duration  of 
the  QA  process. 

Co-develop  the  T&E  plan. 

Co-develop  aircrew  and  operator  checklists  and  questionnaires. 

Supervise  administering  and  collection  of  surveys. 

Co-develop  T&E  final  report. 

Table  2  details  project  milestones. 


Table  3.  Project  Milestones 


Milestone 

Date 

Responsible  Agency 

T&E  Phase  1 

2-3  April  04 

NPS 

T&E  Phase  II 

13-16  April  04 

NPS 

Interim  T&E  Message 

99999 

MAWTS-1 

Final  Report 

1  May  04 

MAWTS-1 

2.2  Personnel  Assignment 

Personnel  assigned  to  this  T&E  are  listed  in  Table  3. 
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Table  4.  T&E  Personnel 


Rank 

Name 

Billet 

E-mail 

Fax 

(DSN) 

Phone 

(DSN) 

Maj 

Weaver,  P.R. 

Project  Action  Officer 

Drweaver(a>nDS. navy. mil 

756-2786 

Mai 

Doty,  C.  R. 

Coordinatinq  Officer 

dotycr(a)mawts1  .usmc.mil 

269-2637 

269-3681 

Maj 

Mowery,  S.  P. 

Aviation  Ops  Analyst 

mowerysD(a)mawts1  .usmc.mil 

269-2637 

269-2684 

3.0  Reports 

The  results  of  this  T&E  will  not  be  released  until  the  final  report  is  signed  by  the  MAWTS-1 
Commanding  Officer. 

4.0  Project  Security 

This  T&E  is  UNCLASSIFIED  FOUO. 
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APPENDIX  B:  POST  EVENT  QUESTIONNAIRE 


UNITED  STATES  MARINE  CORPS 
MARINE  AVIATION  WEAPONS  AND  TACTICS  SQUADRON  ONE 

BOX  99200 

YUMA,  AZ  85369-9200 


TACTICAL  EVALUATION  OF  OPTIMIZATION  METHODS  USED  IN  NEAR  REAL  TIME 

RETARGETING  OF  AVIATION  ASSETS 

March  2004 
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Annex  B:  Questionnaires  and  Data  Sheets  (TO  be  filled  out  by  participant) 

Background.  This  is  a  Test  and  Evaluation  (T&E)  for  the  use  of  the  Rapid  Asset  Pairing  Tool 
(RAPT)  and  NPS  Tool. 

Instructions.  This  survey  is  divided  into  two  sections  that  will  evaluate  RAPT: 

(1)  Effectiveness 

(2)  Suitability 

I.  Background  Information 

Age;  _  Rank  (i.e.  0-3):  _ Service  (i.e.  USMC):  _  Years  of  Service:  _ 

Designations  Held: 
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Questionnaire 


Date _ 

Tool  Assessed:  RAPT  NPS  (Circle  One) 

TST# _ 

Target  Type _ Target  Priority _ 


Evolution 


COI  #6  Timeliness 

Time  Received _  Time  Output  received  (automated  tool) 


Time  DM  determined  option  without  automated  tool _  Time  DM  assigns  Asset 


Remarks 


COI#1  Target  Selection 

Number  of  targets  not  assigned  assets  that  are  a  higher  priority  than  those  assigned 
assets. 

NPS/RAPT _  DM _  Total  Number  of  assets 

assigned _ 

Remarks 


COI#2  Asset  Availability 

Number  of  assets  assigned  that  are  available  based  on  current  situation. 

RAPT/NPS _  DM _ 

Total  assets  assigned _ 

Remarks _ 
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C0I#3  Target  Asset  Pairing 

Number  of  assets  assigned  that  meet  destruction  criteria. 

RAPT/NPS _ DM _  %  Assigned  by  NPS/RAPT _ 

Total  assets  assigned _  %  Assigned  by  DM _ 

COI#4  Mission  Risk  Assessment 

Risk  assessment  to  aircraft  reassigned 

NPS/RAPT _ Prob  of  losing  aircraft/expected  value  of  attrition 

SEAD  required  YES  NO 

SEAD  recommended  by  NPS/RAPT  YES  NO 

SEAD  used  by  DM  YES  NO 

Remarks _ 


COI#5  Persistence 

Number  of  changes  to  ATO  using  RAPT/NPS _ 

Number  of  changes  to  ATO  required  by  DM  actions _ 

Number  of  missions  cancelled  based  on  reassignment  by  RAPT/NPS 
Number  of  missions  cancelled  based  on  reassignment  by  RAPT/NPS 

Remarks _ 


COI#7  Options 

Number  of  reassignment  options  provided  NPS/RAPT 
Number  of  reassignment  options  provided  by  DM _ 

Remarks _ 


COI#8  Tactical  Accuracy 
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WAS  THE  RECOMMENDED  ASSIGNMENT  TACTICALLY  ACCEPTABLE?  YESNO 
Remarks 


THIS  PAGE  IS  INTENTIONALLY  LEFT  BLANK 
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APPENDIX  C:  POST  EVENT  QUESTIONNAIRE 


UNITED  STATES  MARINE  CORPS 
MARINE  AVIATION  WEAPONS  AND  TACTICS  SQUADRON  ONE 

BOX  99200 

YUMA,  AZ  85369-9200 


DECISION  AIDS  FOR  DYNAMIC  RETASKING  OF  AVIATION  ASSETS 

POST  EVENT  QUESTIONNAIRE 


Maj  P.R.  Weaver,  USMC 

Operations  Research  Department,  Naval  Postgraduate  School 
DSN  269-2957  COM  (928)  269-2957 

Maj  S.  P.  Mowery,  USMC 
ADT&E  Department 
DSN  269-2684  COM  (928)  269-2684 


DECEMBER  2003 
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Background.  Marine  Aviation  Weapons  and  Tactics  Squadron  One  (MAWTS-1)  is 
conducting  a  Tactical  Demonstration  (TACDEMO)  of  Decision  Aids  for  Dynamic  Re-tasking  of 
Aviation  Assets.  To  be  filled  out  by  participant. 

Instructions.  This  survey  is  divided  into  two  sections  that  will  assess  the  effectiveness  of 
various  decision  aids  in  support  of  aviation  assets  conducting  time  sensitive  targeting: 

(3)  Operability/Reliability 

(4)  Human  Factors 


II.  Background  Information 

Age:  _  Rank  (i.e.  0-3):  _ Service  (i.e.  USMC):  _ Years  of  Service: 

Designations  Held:  _ 

(SAC,  SWO,  SWO,  DBC) 

Number  of  WTI  operations  participated  in:  _ 
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UNCLASSIFIED 


Qperabilitv/Reliabilitv.  Using  the  scale  below,  rate  each  statement  by 
circling  one  corresponding  number.  Please  include  comments  if  you  have 
suggestions  for  improvements.  Based  on  your  experience,  describe  your  level 
of  agreement  with  the  statement  that: 


1.)  RAPT/NPS  wasable  1  2  3  4  5  6  7 

to  receive  the  ATO  correctly  from  TBMCS. 

Comments: 


2.)  RAPT/NPS  functioned  1  2  3  4  5  6  7 

without  interruption 

Time  interruption  occurred _ Time  interruption  ended _ 

Number  of  interruptions _ 

Comments: 


General  comments  on  RAPT  effectiveness  in  increasing  command, 
control  and  situational  awareness. 
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Reliability  Data  Sheet  (To  be  filled  out  by  system  operator  for  each  error) 

Failure  Number _ 

Time  Failure  Occurred _ 

Time  System  Repaired _ - 

Type  of  Failure  (Circle  One)  Software  Hardware  Network 
Reason  and  Impact  of  Failure 
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To  be  filled  out  by  participant:  (Circle  One)  RAPT  NPS  Decision 

Aid 

Human  Factors.  Using  the  scale  below,  rate  each  statement  by  circling  one 
corresponding  number.  Please  include  comments  if  you  have  suggestions  for 
improvements.  The  following  questions  have  to  do  solely  with  the  ease  of 
understanding  the  display  information  for  the  RAPT/NPS  tools.  Based  on  your 
experience,  describe  your  level  of  agreement  with  the  statement  that: 

STRONGLY  AGREE  STRONGLY  DISAGREE 

1.)  Probability  of  1  2  3  4  5  6  7 

destruction  was  easy  to  interpret. 

Comments: 


2.)  Risk  Assessment  was  1  2  3  4  5  6  7 

easy  to  interpret. 

Comments: 


3.)  Persistence  was  easy  1  2  3  4  5  6  7 

to  interpret. 

Comments: 


4.)  Current  Mission  number  1  2  3  4  5  6  7 

was  displayed  accurately. 

Comments: 


5.)  Target  information  was  1  2  3  4  5  6  7 

easy  to  interpret. 

Comments: 


6.)  Target  priority  was  1  2  3  4  5  6  7 

easy  to  interpret. 

Comments: 


6.)  Associated  threat  1  2  3  4  5  6  7 

information  was  easy  to  interpret. 

Comments: 
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General  comments  on  the  ease  of  interpreting  the  current  dispiay 
information: 
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INITIAL  DISTRIBUTION  LIST 


1.  Defense  Technical  Information  Center 

Ft.  Belvoir,  Virginia 

2.  Dudley  Knox  Library 
Naval  Postgraduate  School 
Monterey,  California 

3.  Marine  Corps  Representative 
Naval  Postgraduate  School 
Monterey,  California 

4.  Director,  Training  and  Education,  MCCDC,  Code  C46 

Quantico,  Virginia 

5.  Director,  Marine  Corps  Research  Center,  MCCDC,  Code  C40RC 
Quantico,  Virginia 

6.  Marine  Corps  Tactical  Systems  Support  Activity  (Attn:  Operations  Officer) 
Camp  Pendleton,  California 

7.  Director,  Studies  and  Analysis  Division,  MCCDC,  Code  C45 
Quantico,  Virginia 
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